INTERACTIVE ELECTRONIC ASSESSMENT SYSTEM AND METHOD - Patent application

Abstract

Systems, methods, and computer readable media related to improving digital human resource technology are provided, where a user has a digital interactive private space, can choose to use the space, conduct interactive assessments of five traits, and collect additional data from others about the five traits about the user, and an automated system can be provided, including automated algorithms that classify and detect patterns from numerical data, for example, that can use correlations between different types of data.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 303,930, filed January 27, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates generally to the field of digital interactive human resource technologies, including employee evaluation, assessment, development, and decision-making training systems. [Background technology]

[0003] Employees are perhaps an organization's most valuable asset. Retaining and attracting employees often requires that organizations invest resources in developing employees. For example, evaluation of an organization's employees is an important and valuable task for better management of employees and the organization as a whole. This can take many forms: evaluation of subordinate employees by their superiors, peer-to-peer evaluations, so-called "360" evaluations whereby personnel are evaluated by superiors, subordinates, and peers, and many other variations. The logistics of conducting the evaluations also vary, with evaluations being conducted in writing, as part of a face-to-face meeting, through electronic means, or some combination of these or other formats. The uses to which the completed evaluations are put are also varied, with evaluation results being used to develop, promote, discipline, reassign, train, or otherwise influence the management of the evaluatees and the organization.

[0004] Traditional performance appraisal processes tend to be provided annually by managers to their employees. Appraisal data can be objective (based on measurable criteria such as productivity and quality standards), subjective (based on manager's perceptions and observations), or a combination of objective and subjective criteria. This type of performance appraisal often suffers from shortcomings, causes frustration for both managers and employees, and fails to achieve its primary goal of improving individual and organizational performance. A common problem with traditional performance management processes is that the process is not adapted to provide a two-way tool for employee growth and development. The problem can be compounded by not having realistic and identifiable numerical measurements that are normalized across organizations to accurately and objectively measure employee performance and growth. Other shortcomings and deficiencies also exist and will be understood by those skilled in the art.

[0005] In the field of interactive assessment applications, such applications are directed to providing corporate management, such as human resources departments, with annual assessments. Such assessments often involve time-limited access to the assessments by employees, such as providing them during an assessment meeting, and only those selected by the company to submit the assessment. Such applications serve to enable assessments for intra-job promotion or advancement by the company, but do not provide features for employees to self-assess and develop improvement measures through interaction with the tool.

[0006] Deficiencies also exist related to providing effective tools for assessment and decision making that are important to an enterprise and having a high performing workforce.

[0007] Thus, there are deficiencies and desired improvements to be addressed across existing digital interactive human resource technology tools and systems. Summary of the Invention [Means for solving the problem]

[0008] According to the principles of the present invention, an embodiment of a system, method, or computer readable medium is provided that implements advances in human resource technology by providing individual users with personal automated private tools that improve the user's personal skills through at least indirect actions and guidance. In some embodiments, a computer implemented method is provided that is a guided interactive self-assessment service. An application can be implemented that can be launched on a server. The application is configured to provide a service to employees of the enterprise. The method can include implementing the application to provide an interactive user interface to employees, the interactive user interface being adapted to provide an automated self-assessment process to the individual users, and implementing the application to establish a private space that the individual users may configure to limit access to the user's private space by other employees, the application being configured to provide the individual users with access to the private space via a browser and configured to invite one or more other users to have access to the individual users' private space.

[0009] The application is configured to allow an individual user to use the application to perform an automated self-assessment, where the assessment may comprise a numerical rating for five attributes and a free-form text description. The application is configured to allow an individual user to use the application to self-select other employees to perform a peer assessment of the individual user on the service, where the peer assessment may comprise a numerical rating for the five attributes and one or more additional free-form text descriptions. The application can be configured to receive and store the numerical rating for the five attributes and the free-form text description.

[0010] The application can be configured to store a library of algorithms, each configured to detect a classification by detecting a pattern in the numerical ratings that satisfy one of the algorithms. The application can be configured to detect one or more of the classifications for an individual user using received numerical ratings for the five attributes from the individual user's self-rating and a plurality of peer ratings for the individual user.

[0011] The application may use a graphical user interface (e.g., by transmitting data to a mobile device or browser, or by launching locally and remotely, etc.) to allow an individual user to view, within the individual user's private space, a structured report based on numerical ratings from self-assessments and peer assessments, the report comprising detected classifications for the individual user, and in conjunction with providing a structured report, may provide the ability to display interactive radar charts that display combinations of the self-assessment numerical rating and other ratings for the individual user, each as a connected polygon shape.

[0012] The application can be configured to store structured reports including the individual users' detected classifications, allow users to return to using the application at a later time to conduct self-assessments and peer assessments again, and configure the application to allow users to review historical data from previous reports within their private space.

[0013] The application may include anonymizing the ratings such that they are not associated with the identity of one or more colleagues. The anonymizing step may include removing identifying information from the ratings received as part of the storing the ratings step.

[0014] The service may be implemented on a company's enterprise network, with which users may interact using, for example, a web browser.

[0015] The application can be configured to provide a list of employees of the enterprise for the user to select from to request a peer rating.

[0016] The service is configured to update the algorithms at the server at another later date, such as to refine an algorithm for a particular taxonomy, or to add new algorithms for new taxonomies.

[0017] The data used in the algorithm may be data from self-assessments or assessments by others on the individual user (e.g., numerical ratings) and may include representative data or data based on its date. For example, an average of multiple numerical ratings on an attribute or other formulas may be used. Patterns are detected in the values ​​by evaluating the data or such processed data and / or interrelationships between data, for example, to detect defined patterns corresponding to classifications. Individuals may have mentors who are provided access to view all or some information within the individual user's private space. Users may select one or more mentors.

[0018] Related non-transitory computer readable media and systems are included in the description herein, as will be understood by the reader. [Brief description of the drawings]

[0019] Various features of embodiments in accordance with the principles described herein may be more readily understood by reference to the following detailed description, taken in conjunction with the accompanying drawings, in which like reference numerals designate like structural elements and in which:

[0020] [Figure 1]FIG. 1 illustrates one embodiment of a system for employee evaluation according to some embodiments of the present invention. [Diagram 2] FIG. 2 illustrates another embodiment of a system for employee evaluation according to some embodiments of the present invention. [Diagram 3] FIG. 3 illustrates a flow process showing communication between an assessment software application and an assessment processing software application according to some embodiments of the present invention. [Figure 4] FIG. 4 illustrates a portion of an exemplary user interface that allows a ratee-user to select options, according to some embodiments of the present invention. [Figure 5A] FIG. 5A illustrates a portion of an exemplary user interface that allows an appraisee to request an evaluation from a peer or mentor, according to some embodiments of the present invention. [Figure 5B] 5B-5C depict a flow process for applying for an evaluation, according to some embodiments of the present invention. [Figure 5C] 5B-5C depict a flow process for applying for an evaluation, according to some embodiments of the present invention. [Figure 5D] FIG. 5D illustrates a portion of an exemplary user interface that allows registered raters to access evaluation forms, according to some embodiments of the present invention. [Figure 5E] FIG. 5E illustrates a portion of an exemplary user interface that allows a non-registered assessor to access an assessment form, according to some embodiments of the present invention. [Figure 6] FIG. 6 illustrates a portion of an exemplary user interface of an assessment software application that allows a user to complete a self-assessment, according to some embodiments of the present invention. [Figure 7] FIG. 7 illustrates a portion of an example user interface of an assessment software application that allows an assessor to create a peer / mentor assessment, according to some embodiments of the present invention. [Figure 8A] FIG. 8A (provided as a group of drawing sheets due to size limitations of constructing 8A, 8A-1) illustrates a portion of a user interface for enabling a ratee user to view submitted assessments and corresponding assessment analyses, according to some embodiments of the present invention. [Figure 8A-1] FIG. 8A (provided as a group of drawing sheets due to size limitations of constructing 8A, 8A-1) illustrates a portion of a user interface for enabling a ratee user to view submitted assessments and corresponding assessment analyses, according to some embodiments of the present invention. [Figure 8B] FIG. 8B (provided as a group of drawing sheets due to size limitations of constructing 8B, 8B-1, 8B-2, 8B-3) illustrates a detailed evaluation report according to some embodiments of the present invention. [Figure 8B-1] FIG. 8B (provided as a group of drawing sheets due to size limitations of constructing 8B, 8B-1, 8B-2, 8B-3) illustrates a detailed evaluation report according to some embodiments of the present invention. [Figure 8B-2] FIG. 8B (provided as a group of drawing sheets due to size limitations of constructing 8B, 8B-1, 8B-2, 8B-3) illustrates a detailed evaluation report according to some embodiments of the present invention. [Figure 8B-3] FIG. 8B (provided as a group of drawing sheets due to size limitations of constructing 8B, 8B-1, 8B-2, 8B-3) illustrates a detailed evaluation report according to some embodiments of the present invention. [Figure 9] FIG. 9 illustrates a flow chart describing the process of identifying and deriving a personality profile of an assessee in accordance with the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Detailed Description of the Invention In accordance with the principles of the present invention, advances in HR technology are provided with interactive computer-implemented tools, software processes, and systems that improve upon existing employee evaluation techniques. There are many advantages and beneficial aspects of the present technology, and several such aspects are discussed first.

[0022] Based on the research conducted, there is a need for a digital interactive tool that greatly promotes employees' self-development and, in particular, greatly improves employees' decision-making skills. Decision-making is a key aspect of employee performance that has historically been underdeveloped and difficult to assess. There are also difficulties in identifying and implementing successful, accessible solutions that benefit employees. The system developed according to an embodiment of the present invention provides an accessible and efficient interactive tool with interactive assessment applications and related features that provide a significant improvement in the field. Through research, certain (limited) attributes have been found to have a close relationship with assessing and improving decision-making skills. These attributes are (1) future perspective, (2) progress measurement, (3) discipline in difficult situations, (4) promptness and boldness, and (5) conscientiousness, which are further described or applied below. The system is configured to implement a software tool for assessment that integrates these attributes and related features to develop the user's decision-making skills (e.g., over time by reusing the tool and learning from interactions). The assessment tool in the workplace can be a useful tool for employees to understand their own strengths and weaknesses and improve the identified strengths and weaknesses through self-awareness. Other applications or attributes (other than decision-making and / or defined attributes) are also envisioned. Preliminary studies indicate that various direct and indirect interactions with the tool provide particularly beneficial improvements to employee performance, while also providing a very convenient and efficient software solution. Thus, in some aspects, the system and software described herein develop a new form of technology that implements a decision-making training system.

[0023] The present invention provides a quantitative assessment of an individual's decision-related skills by analyzing inputs to interactive questions provided by the individual and his / her peers, mentors, and / or family members (for improved accuracy). The quantitative assessment may be used by the individual to develop a training program (e.g., selection of specific training modules) with or without a mentor to improve and / or develop his / her skills (e.g., decision-making skills). For example, if the assessment analysis determines that the individual has reckless traits above a threshold, the system may identify and / or help the individual identify work or training programs for improving the attribute. Specifically, the tool of the present invention quantifies, based on the inputs, the current behavioral characteristics (as interpreted by the system using the data), such as the individual's thinking patterns, gaps in perception, and areas of balance / imbalance, that directly affect decision-making. The questions relate to five factors or attributes: (1) future perspective (as a measure of effectiveness in establishing goals and inspiring others toward them); (2) progress measurement (as a measure of effectiveness in identifying trends, maintaining perspective, learning lessons from the past, etc.); (3) discipline in challenging situations (as a measure of consistency and discipline in tackling difficult tasks); (4) promptness and boldness (as a measure of promptness and boldness in action that takes into account an individual's ability to make tradeoffs); and (5) honesty (not lying, concealing, or falsifying) (as a measure of how an individual communicates honestly to others and their own ability to pursue the truth). An embodiment of the present invention relates to a structured system that uses a combination of software, hardware, data structures, security, and constraints to provide an advanced interactive electronic assessment system. For example, an application can be implemented that provides employees of a company with an interactive user interface adapted to provide users with the ability to self-assess themselves with respect to a set of attributes, preferably a set of five attributes: future perspective, progress measurement, discipline in challenging situations, promptness / boldness, and conscientiousness.The application is preferably configured to establish a private online space (e.g., accessible via a mobile application or website user interface) where individual users may configure to limit access to their private space. The space is where users may view their self-assessment ratings (numeric ratings for each attribute, free text ratings) and may be configured to allow others, specifically one or more invited mentors, to have access to the information in the private space. The application is also configured to allow users to self-select colleagues (e.g., including peers, superiors, etc.), family members (e.g., friends, family, or someone who is not a colleague or mentor), and / or mentors to perform the same evaluation of the user, provide a numerical rating for each attribute, and provide an interface for writing in free form an evaluation for one or more attributes or an overall evaluation of the user. If desired, the system can be configured to limit the length of the text evaluation so that colleagues are forced to refine their thoughts into a short meaningful message, knowing that this will be shared with the user. The application is configured to receive peer and / or mentor evaluations and, when requested (subject to some conditions or limitations, such as whether all peers have completed the evaluation), provide the peer evaluations to the user, preferably in an anonymous state. In other words, the application can delete or remove the identification of the peer and / or mentor before sending the evaluation to the user's private space. In some implementations, the application can implement an operation that reads only data from the evaluation information when reporting the peer evaluation, but does not identify (read the information of) the peer when generating the report on the user. Specific rules can be implemented such that access to the anonymized evaluations (e.g., from peers) is not enabled until at least a threshold number (e.g., 3, 4, 5, etc.) of evaluators have submitted their critiques / evaluations. Other measures to ensure anonymity are also described herein.Privacy and control over access to data can be implemented using tables that define the information that each logged-in user has permission to receive (e.g., based on the table entry, check if the user is a mentor to another user, and based on the permissions in the table, will retrieve and display the user's self and peer assessments in the mentor's private space). Note that while the following disclosure describes the invention in the form of a mobile application, it is not so limited and can be implemented as a website that includes an interactive user interface. It can be implemented as a web application in combination with a mobile application.

[0024] The application can be configured to allow the user to be able to select only coworkers from the enterprise, using the enterprise's enterprise computer system to integrate into the operation and provide only coworkers as selectable options. If desired, the user can be provided with the option to select people outside the enterprise by specifying their email addresses to have them access the application and provide input to the user. Generally, in some embodiments, the self-guided process further reflects that it is the user's choice of coworkers and mentors that is controlled.

[0025] The application may include an analysis and reporting system or software engine and a reporting component. As a matter of background, typically, a corporate evaluation or self-assessment system provides a direct rating on certain specific questions, and the meaning of the combination of evaluation information is left to the subjective understanding of the employee and the human resources department. Conventional tools do not implement advanced features that allow for self-assessment analysis and detection of indirect information as a function of interrelationships of different attributes that are consistent across and / or within multiple companies. For example, with a software engine, once a classification or condition is detected, a system is implemented that stores a library of algorithms, each having a corresponding classification and associated stored data. The software engine is configured to apply the library of algorithms by evaluating a numerical score for each attribute and detecting whether one or more of the algorithms are completed / fulfilled, e.g., detecting a classification. These algorithms may include many levels of complexity, such as being able to detect interrelationships between certain attributes. For example, whether the average of a first and a third attribute results in a higher numerical value than the score of a second attribute. Or, for example, whether the average of the first and third attributes results in a higher numerical value than the score of the second attribute, while the score for the second attribute is lower than both the fifth and fourth attributes, respectively. Complex analyses of many different patterns can be detected and classified from automated data analysis based on a corresponding library of information. The classification through the algorithm and the classification information stored can be used to display the detected classification or other automated assessment to the user through structured reports that are displayed to the user in the application. The reports can be displayed in conjunction with radar charts that provide an interactive radar display. The classification and reporting process provides the same data to (all) users within the enterprise once classifications are detected, and the classifications are provided only to the user's private space and selected instructors (the software limits access to the data).

[0026] The process is structured to have a set of specific (distinct) categories of classifications that the software engine detects and reports, e.g., a predefined set of distinct divisions within the report that are configured to provide a structured view of the report. The system provides an objective evaluation system in a consistent manner for all employees.

[0027] There may be a provider of the application that may maintain a library of classified and retrieved data locally or remotely. The provider may periodically update the tool to refine accuracy and content based on additional employee interactions. With this information, employees are enabled to self-direct their personal development of skills to improve their attributes in a manner that is on par with others in the enterprise. The application may further be configured to be available to the user to perform self-assessments and select peer-request assessments over time (e.g., every three months), and the application may maintain a history of each iteration so that the user may also benefit from viewing historical information via graphs and reports. Web interfaces may be provided for the various features, in combination with mobile applications, and the like. Various embodiments are envisioned, such as individual features may have significant implementation value on their own or in various combinations of the features herein.

[0028] Embodiments of the present invention are directed to a system for implementing an assessment tool for employees of a business entity. In at least some embodiments, the system implements an interactive structure that establishes an employee assessment system that allows employees to perform self-assessments, request assessments from peers, and request assessments from mentors or family members / friends. The employee assessment system provides the added advantage of providing employees with a personalized platform that displays the requested assessments only to the corresponding employee and / or the employee's mentor, and allows employees to identify areas of potential improvement and / or develop strategies for improvement in the identified areas using a specific electronic interface provided in the system. As part of the system, an assessment application (e.g., an assessment software application) is implemented either on a personal mobile device or via a web interface that is configured to provide the user with an interactive assessment tool on the display of the mobile device or other electronic device.

[0029] The evaluation application is configured to send login and password information to a personal identification service, which receives and authenticates the user and provides or sends an electronic security token to the evaluation application, which the evaluation application sends to the evaluation service (e.g., an evaluation processing software application) to obtain authorization to operate on the service. The authentication and the system are configured to use the authorization token to enable access to other services on the enterprise network of the company or provided by the enterprise that employs the user, and to gain access to other services (e.g., to all other services or to join the enterprise network and have access to enterprise applications such as the evaluation processing software application). This process provides efficiency in that the user and the system do not require a separate login for each enterprise application to access the corresponding service. In addition, this can protect anonymity because it can separate the login or name of the evaluator from the interactive process at the evaluation service. The interactive process at the evaluation service can use numeric or alphanumeric identifiers (or other non-descriptive identifiers) for communication (in packets or messages) to the service that would not be visually understood or otherwise easily used to identify the user of the mobile device. Additionally, the use of security processes and tokens establishes highly secure inter-system relationships and connections (e.g., trusted connections using communication and security protocols) based on identity authentication and security tokens used in messaging. The reputation service may be, for example, a reputation processing software application described below.

[0030] The assessment application (e.g., an assessment software application) is configured to provide a user with interactive tools available from one or more display screens displayed by the assessment application on the user's mobile device or as a web application on other electronic devices. The assessment application is configured to display options for requesting assessments from assessors, preparing and submitting assessments for ratees, receiving and viewing assessments (assessment messages) and corresponding analyses.

[0031] The system is configured to provide employees with an interface on their mobile device that provides, or essentially consists of, options for creating and submitting self-assessments, selecting evaluators for peer, family member, and / or mentor evaluations, and viewing the evaluations and various graphical outputs.

[0032] The system is also configured to provide an interface to the rater, which provides or essentially consists of an option to select a text field in which the rater is allowed to type in their own rating in free form using text typing and an assessment rating via the rater's mobile device or as a web application on another electronic device. The system is configured to provide the interface on the user's personal mobile phone by having an application that runs on the mobile phone. This can provide other benefits such as security, speed of use, and reduction of employee hardware resources.

[0033] The system (e.g., available on an employee's mobile phone, which the user carries with them almost all the time) provides for rapid typing of evaluations. The system is configured to provide employee-to-employee evaluations (one-to-one), preferably without restrictions such as seniority or role. The system is also configured to maintain the identity of the evaluator who gave the evaluation anonymously. Furthermore, the system is configured so that the evaluations are only accessible by the appraisee or mentors invited by the appraisee, and not by other employees of the enterprise.

[0034] The mobile application (e.g., a rating software application) is preferably configured to be a lightweight application that imposes minimal processing or storage requirements on a personal mobile phone. The mobile application generates an interactive display screen and provides interactive tools. Information displayed in the mobile application is retrieved after a user opens the application (from a closed state) and launches it on the mobile phone. The mobile application will request information in response to a user selecting an interactive option (e.g., selecting to give a rating and a list of employee names is retrieved). An HTTP GET command can be used for this process. For example, when a user selects an option to view a queue or report of employee-to-employee ratings given in the enterprise, the mobile application responds by transmitting a command requesting messages. The service (e.g., a rating processing software application) responds by sending a set of recent messages that the service received and stored on an enterprise search platform (e.g., system 150 of FIG. 1) implemented in the service. The service can respond by sending a first set, and in response to the user scrolling or searching through the messages, the service can send supplemental messages to enable the user to view additional messages in the feed. Preferably, the service and mobile device operate in this incremental process rather than transmitting all or a significant portion of the message to the mobile application, which can slow down the mobile phone and service and can pose security issues.

[0035] The mobile application (e.g., the assessment software application) uses the mobile device's local volatile memory (e.g., a cache or memory buffer) to store information, such as a queue of assessment messages at the enterprise, on the mobile device in supporting the operation of the application, including the ability to present a display screen containing the received information. Preferably, when a user closes the mobile application on the device, the received information is no longer accessible as it is part of the non-volatile memory, which frees up storage on the mobile phone for other applications.

[0036] The mobile application (e.g., the assessment software application) is configured to communicate with the assessment service (e.g., the assessment processing software application) implemented on the server using two-way data messaging, such as by using HTTP GET and HTTP POST messages. Preferably, email protocol messages or chat messaging are avoided and are external to and / or independent of the mobile application. The mobile application prepares messages for interacting with the service by creating data portions of the messages and associating relevant tags, and further prepares communication messages (or packets) on the mobile device using JSON on the mobile phone, which are transmitted by the mobile device over a wireless network, such as a WiFi or mobile phone network.

[0037] Another application that may be configured to function across or in conjunction with the present system is an assessment processing application. Data collected by the present system from the assessment process may be stored and used in performing analysis by the assessment processing application.

[0038] The implementation of an electronic system for configuring anonymous employee ratings of other employees establishes an existing platform for use by employees of an enterprise. It is conceivable that other systems or purposes will be integrated or combined with the operation of the electronic system (e.g., as a matter of operation or system efficiency). How to implement such integration without adversely affecting the operation and effectiveness of the rating system presents a technical problem to developers. By integrating a second application in the rating system, an enterprise can avoid the need to develop and / or deploy a separate system or software for other purposes. In at least one respect, an embodiment of the present invention integrates an electronic employee rating system into the rating system in a seamless and efficient manner. The rating system is integrated with a user interface for typing in ratings for other employees that uses a specific interface that allows users to freely write text for the rating (e.g., without specifying multiple categories of rating criteria or providing boilerplate text that they can reuse) and specify the rating on a set of attributes.

[0039] The evaluation system implements an interactive tool that can be used by a user to evaluate various behavioral attributes, which are then analyzed to quantify the employee's insightful characteristics (e.g., thought patterns, gaps in awareness, and areas of balance / imbalance). Known techniques or algorithms have not used or implemented such techniques and have been inaccurate in their determination of such behaviors because they do not consider the employee's reluctance or ability to provide accurate responses. The system implements algorithms such as the evaluation of employee behavior that accurately determine the way an employee is behaving toward his / her peers or mentors and the employee's unique perception of the behavior / behavior patterns by applying peer / mentor evaluations and combining them with self-evaluations provided by the employee. The system is configured to apply the algorithms to the feed of evaluation messages (and the data contained therein) and determine the behavior / behavior patterns as quantifiable characteristics. The radar chart can provide a reliable and accurate assessment of employee behavior. The automated classification and report generation process relies on a software engine and underlying rules.

[0040] Implementations and embodiments of such systems, interactive tools, applications, or interfaces (etc.) according to the principles of the present invention are further described below. The discussion includes additional relevant details depending on the embodiment, and advantages and technical benefits are also described or can be understood from the description. The term assessment system is sometimes used in recognition that the described systems involving assessment are directed to electronic interactive systems involving assessment and skill development.

[0041] An embodiment of the present invention is directed to a system and method for employee evaluation. The envisaged system essentially includes an evaluation application and an evaluation processing application. The evaluation application is installed on a mobile device and / or provided as a web interface on an electronic device and can be used by individuals to apply for peer / mentor evaluation, submit self-evaluation, and view evaluation data or results. The evaluation application allows individuals (evaluators) to freely form evaluations in their own unique words. The evaluations are displayed anonymously in the evaluation data and results without indicating the identity of the evaluator. The evaluation processing application provides the data displayed in the evaluation data and results. The evaluation processing application is installed on a server and used to provide anonymous evaluations to the evaluation application. The evaluation application sends the evaluations to the evaluation processing application. The evaluation processing application processes the evaluations to mask information identifying the evaluator and provides the evaluations without the information of the evaluator to one of the recipients and / or data feeds. The evaluation application can also send requests to the evaluation processing application to retrieve data on other feeds. The reputation processing application includes a first database containing sender information and a data cache and readout system that lacks sender information or has the sender information encrypted (it should be understood that in general various combinations are envisioned). The first database is protected with security measures or configurations so that no one (except authorized users) can access it to determine the identity of their raters. The reputation system may also include an analysis application that is used to analyze users' ratings and output various metrics in the form of graphical representations such as radar charts.

[0042] Technical improvements over the prior art may include, for example: Flexible tools that allow users to form their own original true assessments without being restricted to certain terms or selections, or filtered or otherwise modified. Flexible tools may limit the amount of text that assessors can include in their assessments. Software applications and systems capable of processing and providing anonymous original ratings to recipients; A classification engine that uses radar charts and various rule sets to determine the behavioral patterns or personality traits of the assessee; Software applications and systems that may provide limited access to assessees until a sufficient number of assessments have been collected; Software applications and systems that may limit access to certain enterprise resources until an employee takes an action on an evaluation; A private self-nurturing space based on self, peer, mentor, and / or family member assessments and automated analysis; Use of assessments and automated analytics to develop or improve one or more skills of assessees (e.g. decision-making skills); A virtual container system for providing bundled software, libraries, and configuration files for executing the systems and methods of the present invention in a manner isolated from other virtual machine instances / applications / data to preserve the security and integrity of the information contained therein; and ·Structured reporting and visual graphics.

[0043] Other improvements of the present invention will be apparent from consideration of the drawings and ensuing description. The contemplated systems and methods provide a truly anonymous assessment mechanism that can be used to safely and efficiently motivate improvements in decision-making skills, teamwork, and quality.

[0044] Personnel, which may include all or only a subset of the organization's personnel, are divided into two groups: evaluators and assessees. Evaluators are personnel capable of performing evaluations of other personnel, including themselves (i.e., capable of performing self-evaluations). Appraisers are those personnel to be evaluated by the evaluators. There may be one or more personnel in the evaluator group and one or more personnel in the assessee group. In various embodiments, the group of evaluators may fully or partially overlap with the group of assessees. Optionally, the two groups may be separate and distinct. Although this disclosure describes evaluators as belonging to an organization, this disclosure is not so limited and in various scenarios, some evaluators may be personnel who are not employees of the organization (e.g., family members or friends of the assessee who are applied to provide a peer evaluation).

[0045] As used herein, a "self-assessment" refers to an assessment in which the assessor and assessee are the same person, a "peer assessment" refers to an assessment provided by an assessor who is not applied as a mentor by the assessee (e.g., another employee in the same company, a boss, a friend, or a family member) to mentor or train the assessee, and a "mentor assessment" refers to an assessment provided by an assessor who is or will be a mentor selected by the assessee to mentor or train the assessee. Also, a "family assessment" refers to an assessment provided by an assessor associated with the assessment, such as a spouse and / or someone who is not a mentor or a colleague. Although some portions of this disclosure below describe peer, mentor, and self-assessments as examples, this disclosure is not so limited. For example, family assessments can be applied for, provided, and analyzed (including anonymized) in the same manner as peer assessments.

[0046] 1 illustrates one embodiment of a system 100 for submitting, analyzing, and using employee evaluations. The system includes an evaluation or assessment software application (for brevity, referred to herein as the evaluation application) 105 installed on a first electronic device 101(a) and at least a second electronic device 101(b) (collectively, the evaluation portal system) that may communicate with the evaluation software application 105. The first electronic device 101(a) may be used by an appraisee, via the evaluation application 105(a), for example, to select peers and / or mentors, submit evaluations from peers and / or mentors, provide self-evaluations, review evaluations (or corresponding analyses), receive skill development and / or training based on evaluations (self or by mentors), or the like. The second electronic device 101(b) may be used by an assessor (e.g., a peer or instructor) via an assessment software application 105, for example, to receive assessment submissions, provide secure and anonymous assessments, and / or assist assessees in skill development and / or training (e.g., as an instructor), or the like. The system further includes an assessment processing software application 110 implemented on one or more servers (assessment processing system), an analysis software application 115 implemented on one or more servers (assessment analysis system), and a training software application 120 implemented on one or more servers (training system). The electronic device is preferably a handheld, mobile smart phone capable of downloading and installing mobile applications, which may communicate with the server through a mobile phone over a mobile network or other wireless network. Optionally, the electronic device may be a personal computer, a server, a mainframe, a virtual machine, a container, a gaming system, a television, and a mobile electronic device such as a tablet computer, a laptop computer, and the like.The electronic device and the server are each a computer system including a microprocessor and volatile and non-volatile memory for configuring the computer system. The computer system also includes a network connection interface that allows the computer system to communicate with another computer system via a network. The assessment processing software application 110, the analysis software application 115, and the training software application 120 may be implemented on the same server or different servers. The system may include one or more of the aforementioned software applications instead of all three. Each software application may also be used independently or in conjunction with another software application (a software application other than the above three software applications) to enhance or complement the functionality of the other software applications.

[0047] The assessment software application 105 includes a login interface and performs a user authentication process to allow users (registered assessors and / or assessees) to access the assessment software application. It is noted that assessees may be provided with a first type of access that allows them to apply for assessments, review assessments or assessment analyses, and use assessments for skill development / training with or without a mentor. In other words, the first type of access allows assessees to access their private space. On the other hand, assessors may be provided with a second level of access that allows them only to submit the applied assessments (e.g., when the assessor is a peer) and / or a third level of access that allows them to submit the applied assessments and to review the assessments or to collaborate with the assessee on skill development / training (e.g., when the assessor is a mentor). The different levels of access may be configured automatically (e.g., using a rule set) and / or by the assessee based on the role of the user (i.e., assessee, peer, or mentor) using the assessment software application. The application may also be configured to condition such access to users who have performed certain tasks in the assessment system (e.g., submitting an assessment, submitting a self-assessment, submitting a peer / mentor assessment, etc.). For example, the application may require a user to submit a self-assessment and receive a certain number of peer and mentor assessments before a ratee user may view their own assessment and associated analysis for a first time. The system may also be used to unlock access to other information in the application and / or other applications across the enterprise. The system can be configured to use permissions to block an individual user's employer, supervisor, human resources department, or others from accessing the individual user's assessment data (e.g., their own private space of self, peer, and mentor assessments and reports) unless access is granted by that user, such as by a mentor selected by that user.

[0048] The assessment software application is also configured to send reminders or notifications, whether in the assessment software application or through email. In some embodiments, notifications are generated for, for example, applying for an assessment, submitting a requested assessment, viewing an assessment, reviewing training, etc. Notifications may be provided by the assessment software application through push notifications, pop-up windows, sounds, or vibrations, etc., or by the system or EMS sending an email with a reminder.

[0049] A user can enter his / her sign-in credentials from the login interface through an electronic device, and the entered information is transmitted by the electronic device to another computer system via a network. The computer system that receives the user's credentials may be an authentication system 125, which may be part of the system 100. For example, an employee's sign-in credentials may be his / her corporate sign-in credentials, including information that identifies the user as an employee of the company, such as an email address based on the company's email domain, an employee identification number, or other employee identifier. In response to receiving the credentials, the authentication system checks whether they (e.g., email address and password) match the credentials stored in the authentication system (e.g., email address and password pre-stored in the authentication system). The credentials stored in the authentication system may be created and stored in the authentication system, for example, when a person joins the company and the company's human resources department creates an email address and password for the person through the company's computer system. The credentials stored in the authentication system can be updated when a user changes his / her credentials (e.g., password). The authentication system is configured to store the sign-in credentials of employees employed by the company. The authentication implementation is configured to use the authorization token to enable access to services provided on the enterprise network of the company or by the enterprise that employs the user and gain access to the services (e.g., to all other services or to join the enterprise network and have access to enterprise applications such as rating software applications and employee databases). The process provides efficiency in that the user or the system implementing each enterprise application does not require a separate login for each enterprise software application on the user's electronic device to access the corresponding service.At the same time, the process also establishes highly secure inter-system relationships and connections, such as trusted connections using communication and security protocols based on identity authentication and security tokens, to ensure that unauthorized individuals or individuals outside the enterprise do not have access to the enterprise applications. The security implementation provides a simplified authentication process or one single authentication process that can simultaneously verify that a user is an employee of the enterprise and allow the user to access all enterprise applications on the enterprise network. In contrast, conventional systems require separate authorization processes for each procedure, such as an authentication process to verify whether an individual is an employee, a separate authentication process to grant access to one of the enterprise applications after verifying that the individual is an employee, and another separate authentication process to grant access to another one of the enterprise applications after verifying that the individual is an employee. In some applications, employees will use their own personal mobile phone at the employer's premises or remotely, which will implement the mobile application and then connect to the enterprise software platform through an authentication process. For example, the device may be an external device that benefits from enterprise resources through cellular data, another enterprise WiFi, or logging over the enterprise WiFi.

[0050] Optionally, the sign-in credentials may be different from the enterprise sign-in credentials. In such an embodiment, for a first-time user, the user's credentials may be generated by authentication system 125. In some embodiments, a first-time user may also create their own unique credentials from the login interface.

[0051] In response to receiving the authentication information, the authentication system 125 checks whether they match the authentication information stored in the authentication system (stored as a result of the authentication system generating authentication information for a first-time user creating credentials). If they match, the authentication system 125 generates an authorization or security token and transmits it over the network to the electronic device 100. The assessment software application 105 incorporates the token into all of its data to or communications with the assessment processing software application 110. The token is required to communicate with those software applications and is checked and verified by those software applications before the assessment software application 105 can access functionality and data in the assessment processing software 110 and / or employee database 112. In response to a match, the authentication system 125 may generate a user identifier, which can be transmitted over the network to the assessment software application 105. The user identifier may also be included in some or all of the assessment software application 105's data or communications with other software applications and / or electronic devices. The token is required to communicate with those systems and databases and is checked and verified by those systems and databases before the evaluation software application can access the functionality and data provided by those systems and databases. The user identifier may be a code or number that only the system or software application knows as the user's identity. For example, the user, another user, or a technician does not know the user's identity when the code or number is presented to that individual. If the authentication information does not match, the user (or electronic device) is blocked from accessing the evaluation software application 105 and other software applications and an invalid message is transmitted to the user.

[0052] Similarly, the authentication system 125 may control access to the assessment software application 105 by other unregistered users (e.g., non-employee users) via the second electronic device. For example, the assessment software application may provide a token to the second electronic device to enable access to the unregistered user for purposes of providing an assessment regarding the ratee, when such an unregistered user is selected by the ratee as a ratee.

[0053] Once access to the assessment software application 105 is granted to the assessee, the assessment software application 105 provides a user interface (e.g., on the first electronic device 101(a)) that allows the assessee to perform several functions, such as requesting an assessment from a peer or mentor, completing a self-assessment, viewing assessment progress, viewing assessment results, working on skill development or training, or the like. For example, FIG. 4 illustrates a portion of a user interface 400 that allows the assessee user to select an option to proceed, for example, to a self-assessment user interface (using command 410), an assessor selection user interface (using command 412 or 414), viewing assessment results (using command 416), or the like. As shown in FIG. 4, the user interface may be configured to be shown as a progress bar 450 that illustrates the assessee's progress during the assessment process. A command may be a button (physical or digital button), an icon, a tab, a window, or the like that allows the user to activate functionality of the software application or computer system.

[0054] FIG. 5A illustrates a portion of a first user interface 500 that allows an assessee to request an assessment from a peer or mentor, either by searching for an assessor who is already a registered user from a list of registered users (501) or by providing an email address for the assessor (502). This screen may be displayed in response to the assessee user clicking on the assessor selection command (412 in FIG. 4). The list of registered users may be employees of a company (the same company). FIGS. 5B and 5C illustrate flow processes 501 and 502 showing the communication between the assessment software application and the second electronic device 101(b) for requesting an assessment from a registered and unregistered user, respectively. FIG. 5A illustrates how an assessee user 505 requests a list of registered users (e.g., within an organization) 510 from a user interface of the assessment software application 515. In response, the evaluation software application sends a list of registered users 520 (e.g., from an associated database) to the ratee user and receives an evaluator selection 525 from the ratee user. The evaluation software application 515 is configured to communicate with an employee database (e.g., FIG. 1, 112) and retrieve employee identification information from the employee database. The evaluation software application 515 can display all employees of the company (regardless of position, title, or role) or a limited set of employees, such as employees determined to be employed at the same location, employees who are part of the same department, employees who are part of the same subgroup (or set) as that of the ratee, the ratee's supervisor, or the like. The evaluation software application then utilizes the rater user identifier to retrieve further information 530 (e.g., name, email, etc.) about the selected rater. The information 530 may be retrieved from a database that stores information about previously registered users. Specifically, the employee identification information may include the employee's name and an identifier for each employee to be used by the system to make messages available to the appropriate person.For example, in response to selecting "Michael" from the employee list, the evaluation software application determines that "Michael" is associated with identifier "0020" and presents the message to identifier "0020" so that Michael may view the message. The identifier may be an internal identifier used by the system and the employee database to identify the employee. This identifier may be referred to as a recipient identifier. Updates to the employee database also update the employee list in option 520 accordingly. The evaluation software application may then send a message 535 containing the evaluation invitation template to a messaging service 340 (e.g., an email service) for transmission to the selected evaluator 350. The evaluation invitation may include, for example, an identification of the evaluee user, information related to how the evaluator may submit an evaluation (e.g., by accessing the evaluation software application), or the like. Such an evaluation invitation may be automatically associated with the evaluator's account with the evaluation software application. Specifically, in some embodiments, the employees on the list are also individuals who have access to the evaluation software application. If the employee has not installed the application on his electronic device using his sign-in credentials, messages sent to him will be stored on the electronic message processing system and the employee can view them after he installs the application. In an embodiment, the list may not show individuals who have not yet installed the application (only those who have installed the application are shown).

[0055] For clarity, in FIG. 1, for example, the application is illustrated as a mobile application, but the diagram could also illustrate a web server implementation by having a web server implemented as part of the application environment and running the service application. The assessment application could be configured to be implemented as software as a cloud type application on a browser on a mobile phone or other type of device. The application provided by the web server is provided over a communications network, which may include the Internet. Other web or type implementations are also envisioned.

[0056] For example, FIG. 5D illustrates a portion of an exemplary user interface of a registered assessor's assessment software application illustrating a submitted assessment as an assessment pending request 510 .

[0057] FIG. 5C illustrates how a ratee user 505 requests an evaluation 510 from an unregistered user via a user interface of an evaluation software application 515. For example, the ratee user 505 may provide an email address of the unregistered user. In response, the evaluation software application generates a secure unique token 520 for the received email address and then sends a message 525, including the evaluation invitation template and the generated token, to a messaging service 330 (e.g., an email service) for transmission to the selected evaluator 340. The evaluation invitation template may include, for example, the identification of the ratee user, information related to how the evaluator may submit an evaluation (e.g., a URL link to access the evaluation software application), login credentials (e.g., based on the generated unique token), or the like. For example, FIG. 5E illustrates a portion of an exemplary user interface of an invitation for an unregistered user to provide an evaluation, including an embedded link 510 to access the evaluation form without having access to the evaluation software application itself.

[0058] FIG. 6 illustrates a portion of a user interface of an assessment software application that allows a user to complete a self-assessment. As an example, a ratee user may click on a self-assessment command 410 to access a self-assessment form user interface. FIG. 6 illustrates a portion of a user interface of an assessment software application that allows a user to create and submit a self-assessment. A self-assessment creation area 600 includes a free-form message creation area 605 that allows a user to create an assessment message in their own words with respect to one or more assessment criteria or attributes (e.g., "future outlook" 625 shown in FIG. 6), a multi-level rating scale 610 that allows a user to select a numerical assessment rating, a save command 615 to save an incomplete self-assessment draft, and a submit command 620 to submit a completed self-assessment for transmission to another computing device and / or for local storage. Optionally, a separate self-assessment creation area may be displayed for each assessment criterion in succession until the ratee user completes the self-assessment for each assessment criterion. The ratee user may also access their past self-assessments. In various embodiments, if the user has completed only one or two of the three fields and / or has not completed all fields of all available evaluation criteria, the submit command 620 may be disabled and the user will not be able to submit the self-assessment. Preferably, only five or a small set (such as four to six) of the attributes are used, and the specific attributes are as specified. The time the self-assessment is created (i.e., when the user selects the submit option 620) is also recorded by the evaluation software application.

[0059] The free-form text area is an area in which a user may type a message by typing on the QWERTY keyboard of the electronic device in his own words. The user may also type such messages in other manners, such as by speaking or using gestures to convey his own words to the electronic device. The message the user types is what is transmitted to the evaluation processing software application. The user's ability to type his desired words is not restricted or limited by the evaluation software application (e.g., when forced to select only certain options or predefined messages), and the user can type any words he desires. Optionally, the free-form text area may be associated with a maximum allowed message length. The numerical evaluation rating may be constructed, for example, as a 1-4 scale, with 1 being "Below Expectations", 2 being "Meets Expectations", 3 being "Exceeds Expectations", and 4 being "Greatly Exceeds Expectations". Multi-level and type rating scales are also envisioned. In some aspects, the assessments are configured to be short by design, such as having only a small set of numerical ratings and a textual assessment (limited in size) with few or no other questions or prompts (maybe just one question) that are required to proceed through the assessment without error.

[0060] FIG. 7 illustrates a portion of a user interface of an assessment software application that allows an assessor to complete a peer or mentor evaluation. As an example, an assessor (peer or mentor) may access the user interface 700 in response to clicking a URL link included in an email received with an invitation to provide an evaluation (for unregistered assessors) or by accessing the assessment software application (for registered assessors). FIG. 7 illustrates a portion of a user interface of an assessment software application that allows an assessor to create and submit a peer / mentor evaluation. Similar to FIG. 6, the evaluation creation area 700 includes a free-form message formation area 705 that allows a user to form an evaluation message in their own unique words with respect to one or more evaluation criteria (e.g., “future outlook” 725 shown in FIG. 7), a multi-level rating scale 710 that allows a user to select a numerical evaluation rating, a save command 715 to save an incomplete evaluation draft, and a submit command 720 to submit a completed evaluation for transmission to another computing device and / or for local storage. Optionally, a separate assessment creation area may be displayed for each assessment criterion in succession until the assessment is completed by the assessor for each criterion. In various embodiments, if the assessor completes only one or two of the three fields and / or does not complete all fields for all available assessment criteria, the submit command 720 may be disabled and the user will not be able to submit the assessment. The time the assessment is created (i.e., when the user selects the submit option 720) is also recorded by the assessment software application. The system can be configured to make the peer and mentor assessment process identical and provide a comparative reference (i.e., have the same attributes, numerical ratings, and / or free text areas). Optionally, the self-assessment may be identical but without text input.

[0061] FIG. 8A illustrates a portion of a user interface for allowing a ratee user to view submitted assessments and corresponding assessment analysis. As an example, the ratee user may click on the View Assessments command 416 (of FIG. 4) to access a user interface for viewing the assessments (free-form and numerical assessment ratings, and corresponding analysis). The user interface 800 shows the ratee identifier 810, the average score 815 he received in each assessment type (802: self / peer / mentor), and a summary 820 of the assessments he received per assessment criterion (801) in each assessment type. The assessment software application also provides an option that may be selected to view each submitted assessment in detail. A graphical output (e.g., radar chart) 850 is also displayed and is discussed in more detail below. Other information, such as identified behavioral patterns or behaviors, associated explanations, suggestions for training or skill development, and / or links to training, may also be included within the user interface. The user interface 800 is interactive, allowing the assessee to filter, summarize, expand (e.g., by hovering over), change, or otherwise manipulate the displayed information contained within the graphical output.

[0062] In various embodiments, if the self-assessment and / or the submitted assessment is not completed, the view assessment command 416 may be disabled and the user will not be able to view the user interface of FIG. 8A. Only the assessee can view the completed assessment and the corresponding assessment analysis, and no other users or employees using the assessment software application can access it (e.g., the assessment and the corresponding assessment analysis are available only as a private space on the assessee assessment software application). Optionally, the assessee may invite or allow a mentor to view at least a portion of his / her private space in order for the assessee to collaborate with the mentor in reviewing his / her assessment and generate a plan for self-development (e.g., training). This can be implemented by the mentor being provided with access to the data, with the application displaying information to the mentor in his / her interactive space generated by the application. The assessee can only view the assessment from the assessment software application installed on his / her electronic device, into which he / she needs to log in (see login and authentication process described above) if he / she has not already logged in or is logged out in order to use the application.

[0063] FIG. 8B illustrates detailed reports including, but not limited to, an assessment progress bar 861, an overall score as a radar chart 862, an analysis report 863 (including subparts such as 863(a), 863(b), and 863(c)), and sections such as a view assessment section 864 that allows the user to view the merged ratings and free text ratings for each of the attributes. These are described in more detail below. It is emphasized that the system is configured to perform a process and generate a report in a structure in response. In particular, subparts 863(a), (b), and (c) are configured to report on a corresponding group of algorithms. In the first group, which in this example is a relational gap in recognition, the group of algorithms is configured to analyze the gap between the data between self and peers and between self and instructors. This may be a geometric domain type algorithm to determine geometric or domain differences. The system reports on the identified classification and generates displayed information corresponding to the specific classification. The second group, disparity, corresponds to using an algorithm configured to evaluate each of the self-, peer-, and instructor-generated data and identify data pattern classifications for each. The third group corresponds to an overall score, and the algorithm is configured to generate an overall score. As shown in the example, there is a score generated from the five numeric ratings by the instructor, which are determined using clustering and averaging. This structure can further benefit the user's process.

[0064] In various embodiments, the ratings displayed may be anonymized, using methods discussed below, such that the ratee is not provided with identifying information related to the rater associated with each individual rating. Optionally, peer and / or family ratings may be anonymized, while instructor ratings are not. A trusted anonymity feature is implemented.

[0065] The assessment software application 105 is configured to generate an assessor identifier that identifies the assessor, a message identifier that identifies the assessment message formed, and optionally a time identifier (time information) that identifies when the assessment is sent (e.g., date and time), and an assessment type identifier (e.g., peer assessment, mentor assessment, self-assessment, etc.). The system may also identify the assessee associated with the assessment and create an assessee identifier. For example, the assessee may be identified using a token associated with the assessment and sent to the assessor with the assessment invitation (see, for example, Figures 5B and 5C). These identifiers may be generated in response to the assessor user selecting a submit command for the assessee (after the assessor user completes the assessment message and selects a numerical assessment rating). After the user selects the send command, the assessment software application 105 transmits the assessment (including the assessment message and the rating) and the generated identifier via the network connection interface of the electronic device 100, over the network, to another computer system (e.g., a server on which the assessment processing software application 110 is implemented).

[0066] With respect to the assessment processing software application 110, the server on which the assessment processing software application 110 is implemented is configured to receive the assessment and the generated identifier over a network via the server's network connection interface. The assessment processing software application 110 also checks whether an authorization token is present and verifies whether the token is valid to associate the received assessment with the assessee, to determine the identity of the assessee, and / or to verify that the assessment was submitted by the assessee. The assessment processing software application 110 includes two environments: an application environment 130 and a reporting environment 135. Received data and tokens are input from the application environment 130.

[0067] In some implementations, the anonymity aspect can be implemented using one database, such as the first database 145, software processes can be used as software modules, and / or permission information in tables can be stored to limit the information provided in the peer evaluation report to maintain anonymity. In others, additional or alternative anonymity features can be implemented.

[0068] In one embodiment, the application environment 130 includes a service (or service application) 140 that processes the received data, a first (primary) database 145 for storing the received data, a data cache and retrieval system 150 for storing a portion of the received data, and a second database 155 for storing a portion of the received data. In response to receiving the data, the service 140 stores the received data in the first database 145. The service 140 may store the entire received data, i.e., the evaluation and the generated identifier, in the first database 145. In other words, the service 140 stores what is received by the application environment 130 in the first database 145. In some embodiments, the service 140 stores at least the evaluation (evaluation message and rating), the evaluator identifier, the evaluation type identifier, and the evaluee identifier in the first database. Other combinations are also envisioned. The data stored in the first database is raw data, meaning that the received data is identical to the stored data or that there are no additional procedures applied to remove, add, or modify (e.g., encrypt) any data in the received data other than those fundamental or inherent in the operation of the software application and server, such as decoding, analog / digital conversion, etc.

[0069] The service 140 also replicates the received data, removes or erases the rater identifier from the replicated data, and stores the remaining data in the replicated data (or the replicated data without the identifier) ​​in the data cache and retrieval system 150. The service 140 may replicate the received data as the data is received or from data in the first database 145. The data 150 in the data cache and retrieval system 150 lacks an rater identifier and optionally includes an assessment type identifier. The data 150 in the data cache and retrieval system 150 can only be retrieved by the ratee (i.e., the ratee's assessment and rating) using the ratee identifier and pursuant to the ratee's authentication.

[0070] In addition, the rater identifier in the first database 145 may be replicated and stored in the second database 155. The second database 155 is not linked to the data cache and retrieval system 150. The database 155 and the system 150 are two separate and independent systems. Replication can be accomplished through asynchronous jobs. The rater identifier in the second database 155 may further be encrypted. The time information in the first database 145 may also be replicated, diluted, and stored in the second database 155. For example, the time information may indicate the exact date and time the assessment was submitted, such as 2:00 PM on May 2, 2018. This information may be generalized or downgraded (e.g., to a lower resolution in time) such that the modified time information only indicates that the assessment was submitted on May 2, 2018, in the first week of May of 2018, or in May 2018. The time information may be modified by eliminating only the time portion (e.g., 2:00 PM), by eliminating the time portion and changing the date portion to a lower resolution (e.g., from May 2 to the first week of May), or by changing the time portion to a lower resolution (e.g., from 2:00 PM to PM) and the date portion to a lower resolution. Similarly, the rating type may also be eliminated to obtain a lower resolution. A higher resolution in time means that the time information is more specific, while a lower resolution in time means that the time information is less specific. Similarly, the rating type identifier may also be eliminated to obtain a lower resolution. A higher resolution in rating type means that the rating type information is more specific, while a lower resolution in rating type means that the rating type information is less specific. Encryption and dilution are used to anonymize the identity of the rater. The duplication, encryption, and dilution may be performed under instructions given by the service 140. The time / rating type information in the system 150 is stored in the original time resolution.

[0071] For clarity, the process performed by the service 140 refers to one or more of the aforementioned storing (with respect to the first database 145), replicating, removing, and storing (with respect to the data caching and retrieval system 150) steps. They may further include replicating and storing the assessor identifiers in the second database 155.

[0072] The reporting environment 135 accesses the data in the data cache and retrieval system 150 and performs the analysis.

[0073] As an alternative to removing the rater identifier from the raw data before storing it in the data cache and readout system 150, the assessment processing software application 110 can be implemented using asymmetric encryption. In this embodiment, the application environment 130 is equipped with a public key that is configured to encrypt the rater identifier. The application environment 130 encrypts the rater identifier differently (e.g., with different codes) for each assessment submitted by the same person, but can understand that the different codes are associated with the same rater identifier or person. For example, the application environment 130 can encrypt the rater identifier of assessment message A with XXXX and the rater identifier of assessment message B with YYYY when both assessment messages are submitted by the same person. After encryption, the encrypted rater identifier and other data (e.g., assessment messages, ratings, and other identifiers) are stored in the data cache and readout system 150. The reporting environment 135 accesses and uses the data in the data cache and readout system 150 to perform assessment analysis. The reporting environment 135 is equipped with a private key to decrypt the encrypted rater identifier so that it can perform a rating analysis on the user corresponding to that identifier or provide a sentiment analysis for that user. The encrypted rater identifier can only be decrypted using the private key.

[0074] Each environment and service may be referred to as a system or subsystem within the assessment process software application.

[0075] In an identifier removal embodiment or asymmetric encryption, the data in the first database 145 and the data cache and readout system 150 may be stored as a table. The received data may be converted into a different data structure, such as a table, by the service 140 before it is stored, or by the database 145 or the data cache and readout system 150 after the service 140 provides the data. For example, the service 140 (or the first database 145 / data cache and readout system 150) may create a table by separating the assessment message, identifier, and rating into at least two categories and store each category as a table. The assessment message, assessment type identifier, ratee identifier, message identifier, and rating may be categorized in one category and stored in one table, and the rater identifier may be categorized in a separate category and stored in another table. The two-table format is stored in the first database and the data cache and readout system. As discussed above, the rater identifier may be removed or encrypted before being stored in the data cache and readout system. Thus, the two tables in the first database include the rater identifier and the two tables in the data cache and readout system either lack the rater identifier or have an encrypted rater identifier. The reporting environment 135 may access the data in the two-table format. The rater identifier in the second database 155 may be replicated from the data in the two-table format in the first database 145.

[0076] The data cache and retrieval system 150 is used by the service 140 to retrieve the necessary data in response to a request sent by the assessment software application 105 (e.g., when a ratee uses the assessment software application 105 upon authentication of the ratee login credentials). The data cache and retrieval system 150 is used by the service 140 to keep the identity of the rater anonymous since the rater identifier is either unavailable or encrypted. When the analysis requires this, the rater identifier in the second database 155 is used or the encrypted rater identifier is unencrypted only in circumstances limited by the reporting environment 135. Furthermore, the data cache and retrieval system 150 is used by the service 140 so that ratees may access only their own assessments and ratings without the rater identifier.

[0077] The data cache and retrieval system 150 (or retrieval system for simplicity) is a system that includes a cache mechanism, an index mechanism, and a database. The cache mechanism may be based on Redis or memcached technology and is used to cache data in the retrieval system 150, especially data that has been previously found or retrieved. For example, when the service 140 queries the database in response to a request sent by the evaluation software application 105 and the retrieval system 150 returns the results, the results are stored in the cache for a period of time so that when they are called, the results can be accessed quickly (e.g., retrieved in milliseconds instead of seconds). The index mechanism searches and indexes the data in the retrieval system 150 and saves the index in the index mechanism. The index mechanism allows for the data in the retrieval system 150 to be retrieved quickly. The data retrieved by the index mechanism may be either data that has not been retrieved before or data that has been stored beyond the period of time allowed by the cache and is cleared from the cache. The indexing mechanism may be the default mechanism used by the readout system 150 to retrieve data, and the caching mechanism may be used by the readout system 150 if the data to be retrieved is already in the caching mechanism. With respect to a database, the caching mechanism and the indexing mechanism operate on the data in the database to retrieve the required data. For example, the indexing mechanism may invert index each rating message and rating in the database. When a search is performed by the reporting environment 135 or the rating software application 105, etc., for the rating types in the rating messages stored in the readout system 150, the index format allows the environment 135 or the software application 105 to quickly identify and retrieve those messages. Other techniques and platforms related to searching are also envisioned.

[0078] The reading system 150 also includes two security measures or configurations to protect the identity of the rater as well as the rating information of the ratee. One security measure is that the rater identifier is used by the reading system 150 to find the rating messages and ratings stored in the reading system 150 that the rater gave. The identifier only allows that particular user (through his / her own rating software application and reading system) to search for the ratings he / she gave. The user (and the rating software application and reading system) cannot use it for other purposes, such as searching for ratings that other people gave about the ratee (except when the user is the ratee himself / herself) or the identities of other raters and their ratings. Similarly, the ratee identifier is used by the reading system 150 to find the rating messages and ratings stored in the reading system 150 that correspond to the ratee. The identifier only allows that particular user (through his / her own rating software application and reading system) to search for the ratings he / she received (without the rater identifier, i.e., anonymously). The user (and the rating software application and readout system) cannot use this for other purposes, such as to look up ratings that other people have given on other ratees or the identity of the rater. Another security measure is for the readout system 150 to be configured to work with a system that provides or is implemented with public-private key cryptography. When the system does not involve asymmetric cryptography, the readout system 150 can provide such functionality and the system can operate as described above (alternative embodiment). When the system already has this feature, the encrypted rater identifier is indexed by the readout system and used to quickly find anonymously the ratings (also indexed) given by the individual of that identifier.

[0079] The data caching and retrieval system 150 is referred to as an enterprise search and indexing engine that is capable of retrieving information from many different databases, including the database within the system 150 and other databases. The system 150 provides faster and more responsive capabilities in the context of a rating system. For example, the user interface of the present system simplifies the rating information to be stored, and the system 150 can quickly index, store, search, and retrieve data because of the simplified structure. In systems other than rating systems or rating systems that do not provide a simple user interface, the amount of information received from, for example, Word or PDF documents, can be significant, and therefore using the system 150 may not be as fast and efficient in indexing, storing, searching, and / or retrieving data.

[0080] The first database 145, which stores the raw data or all received data (including the assessor identifier), is constructed with security measures or configurations to prevent unauthorized access. The first database 145 may be implemented in a separate private network and behind a firewall different from the private network and firewall in which the assessment processing software application (excluding the first database 145) is implemented. For example, accessing the first database 145 may require a login process or credentials different from those of the assessment software application (separate portal). The first database 145 may be known as a production database server. Security measures or configurations can be implemented at two levels: application level and database level.

[0081] In one embodiment, the first database 145 is highly restricted (application level) so that only certain software and service applications may access the first database 145. The first database 145 is also highly restricted (database level) so that only certain individuals who have full access to the first database 145 or who have access to at least the assessor identifier in the first database 145 may access the first database 145. At the application level, the first database 145 may grant permissions to the evaluation software application 105 and the service application 140 (and only those applications) that allow them to write to and read from the first database 145. The evaluation software application 105 and the service application 140 may be configured so that they write data to the first database 145 and read data from the data cache and retrieval system 150. At the database level, only the first database creator and individuals approved by the first database creator have access. The creator and authorized applications and people can view or read the raw data, including the rater identifiers and the actual ratings, from the first database 145. The raw data is not encrypted. The creator and enabled individuals access the first database 145 through a two-step security process. First, they are required to log into a private network on which the first database is located that is different from the private network on which the rating processing software application (excluding the first database) is located. Second, their credentials are checked by the first database 145 to see if they have permission to access data in the first database 145 if they wish to access that data. In some embodiments, the second step may involve submitting another login credential that is different from the first step. Both are privileged access and require permission from the creator.Only individuals who pass both steps can access or view the raw data in the first database 145. Some individuals may be granted only the first privileged access so that they may work on the first database 145 to modify or update the hardware and software of the first database 145 or for other purposes such as testing and network-related issues. However, those individuals do not have access to the raw data in the first database 145.

[0082] A database management software application 160 is implemented to interact with the first database 145. The database management software application 160 may be accessible only by the first database creator. The database management software application 160 provides a user interface that allows the first database creator to grant first database access permissions to software applications (e.g., the assessment and service applications 105, 140) and individuals (e.g., a first privileged access granted user and a second privileged access granted user). A second privileged access granted user refers to a user who is granted both the first and second privileged access.

[0083] A user access log system 165 (or software application) is also implemented to interact with the first database 145. The log system 165 may be part of the first database 145 or the assessment processing software application 110. The log system 165 is configured to record all authorized user accesses to the first database 145, including the activities of the first database creator. The log system 165 may record activities that occur under the first privileged access and the second privileged access. For example, the activities that occur under the first privileged access may include the identity of the authorized user, the hardware, software, data, and network that the authorized user modified, the date and time of the authorized user's actions, and other information. The activities that occur under the second privileged access may include the identity of the authorized user, the raw data accessed by the authorized user, such as the assessor identifier and the actual assessment, the corresponding assessee identifier, the date and time of the user's actions, and other information. The log system 165 is configured to be accessible by a limited number of users, such that the first database creator or other professionals (e.g., technicians, engineers, network administrators, IT specialists, etc.) may check whether anyone (including the creator) has accessed the data in the first database 145. Because the system is configured to keep the identity of the assessors anonymous, the log system 165 is designed to check whether there has been any human access to the data in the first database 145 that could give unauthorized access to the identity of the assessors, unauthorized access to the assessment information, etc., when an authorized individual is not supposed to have access to this information.

[0084] Another embodiment of security measures or configurations for the first database 145 can also be implemented. The second embodiment is also implemented at the application and database levels. At the application level, this involves two entities: the software and service applications, and the application administrator. The software and service applications (or one of the applications) are provided with a security key or hash that allows them to encrypt the ratings and access the first database 145 (e.g., store the encrypted ratings in the first database and retrieve the encrypted ratings from the first database) only for authorized users (e.g., corresponding ratees). The same security key may also allow the software and service applications to decrypt the retrieved ratings, or the software and service applications may be provided with a second key to decrypt the data (so that the actual messages and ratings as well as the rater identifiers can be seen or otherwise available). The application administrator, e.g., a professional who manages or controls the rating software and service applications, may also have the decryption key and may be the only individual with such a key. The first database 145 and the database administrator (described below) do not have the decryption key and cannot break the encryption.

[0085] At the database level, this also involves two entities: the first database and a database administrator. The first database 145 is equipped with a security mechanism that can check whether a user attempting to access the first database 145 has the necessary security key (the first database 145 itself may also have such a key). The database administrator, e.g., a professional who manages or controls the first database 145, may be the only individual provided with such a key. The security key for the first database is different from the encryption / decryption key at the application level. Software applications and service applications as well as application administrators (described above) do not have the security key for the first database 145 and cannot access the first database 145.

[0086] Based on the above, accessing and decrypting the ratings in the first database 145 requires both an encryption key and a security key for the first database so that the actual messages, ratings, and rater identifiers can be seen or otherwise available. Both keys may be generated and provided to their respective administrators at the beginning when the second embodiment of the security measures or configuration is installed. A platform may be implemented with the second embodiment that allows administrators to submit their keys. Administrators can access and decrypt data in the first database 145 from the platform. To further enhance the security level of the second embodiment, application administrators and database administrators may be limited to individuals who hold the highest positions (in terms of job title) or executive-level positions within an organization, or who may make responsible decisions on behalf of the organization (e.g., Chief Executive Officer, Chief Financial Officer, and Chief Technology Officer). For example, an application administrator may be a CTO and be provided with a decryption key, while a database administrator may be a CEO and be provided with a security key to the first database. This therefore requires both the CTO and the CEO to submit their individual keys into the platform in order to access and decrypt the data in the first database (to see the actual ratings and the identities of the rating providers). Thus, the first database creator may not have either key and will not be able to access or decrypt the data in the first database 145, even though he designed and installed the first database and its technology (e.g., its security measures or configuration).

[0087] The second embodiment may be referred to as a foolproof security model with distributed security access (e.g., between application administrators and database administrators). Security measures or configurations based on a combination of the above two embodiments are also envisioned. Security measures or configurations based on a combination of one or more features of the first embodiment and one or more features of the second embodiment are also envisioned.

[0088] The evaluation software application 105 and the analysis software application 115 may not be able to access the first database 145. Both applications may be directed to a data cache and retrieval system 150 to obtain the required data.

[0089] Implementing the services, databases, data cache and retrieval systems, and security measures discussed above allows the evaluation system to maintain the identity of the evaluator anonymous while still being able to perform analysis on the evaluations given by the evaluators in a limited and private manner (without unauthorized access to the evaluator's identity). The embodiments of the present disclosure in particular ensure the highest level of access restriction and data protection for information identifying the evaluator, and allow the evaluee to only access their own evaluation. The first database and related applications and systems built with the above features improve the operation or security operation of the first database. They may also, in some embodiments, improve the storage and organization of received information in the evaluation processing software application.

[0090] However, such a level of security may not be implemented in some embodiments. For example, control over what data is accessible to or shown to a user may be controlled using the above-referenced tables that define user permissions, which may not involve application-level or database-level security. This may also not involve, for example, implementing the more complex anonymization processes described.

[0091] FIG. 3 illustrates a flow process 300 showing communication between an evaluation software application and an evaluation processing software application. In particular, FIG. 3 illustrates how submitted evaluations are stored within the application environment of the evaluation processing software application and how submitted evaluations are provided to the evaluation software application. A ratee user 305 requests an evaluation (310) from an evaluator (315) from a user interface of an evaluation software application 320, where the request is associated with a ratee identity. The evaluation software application (320) sends an evaluation invitation (325) to the evaluator (315), also associated with the ratee identity and optionally including a token for an unregistered user. The evaluator 315 types a requested evaluation 330 (including an evaluation message and a rating) from a user interface of the evaluation software application 320. In response to the evaluator selecting the send command, the evaluation software application sends a request or request payload 335 containing the evaluation, the evaluee identifier, the evaluation type identifier, and the evaluator identifier (collectively referred to as evaluation data 340) to a service application 345 of the application environment. The service application 345 may then store the evaluation data 340 or the raw data in a suitable format in a first database 355 as data 360 (step 350). The first database 355 then responds to the service application 345 with either a success or error message indicating whether the saving of the raw data was successful or unsuccessful, respectively (step 365). The service application 345 may also replicate the same raw data, remove the evaluator identifier from the replicated data, and store the replicated data without the evaluator identifier in a data cache and retrieval system 375 (step 370). The replicated data without the evaluator identifier may be stored as 380.The data cache and readout system 375 then responds to the service application 345 with either a success or an error message indicating whether the saving of the replicated data without the rater identifier was successful or unsuccessful, respectively (step 385). Once the saving of the data without the identifier is successful, the service application 345 provides the data that the user selected to send but without the rater identifier to the evaluation software application (step 390). For example, the service application 345 may provide the data to an analysis system for analysis and / or to the evaluation software application for display to the ratee user 305. The evaluation software application, upon authentication, shows the ratee the evaluation he or she received and the corresponding analysis. No other users are allowed access to the ratee's evaluation or analysis. Specifically, the service application 345 determines a ratee identifier corresponding to the ratee, retrieves the evaluation message and rating using the determined ratee identifier from the readout system, and provides the retrieved evaluation message and averaged rating to the ratee. The assessment software application continually updates the data to show the most recent assessment (and previous assessments) received by the assessee, e.g., the assessment submitted in step 310. In some embodiments, the system is configured to use HTTP GET and HTTP POST messages to send and receive assessment messages to and from the server.

[0092] It should be noted that while the above description describes keeping rater identities anonymous, the present disclosure is not so limited and in some embodiments the system may not use such anonymization feature. For example, the rater may instruct the system not to keep their identity anonymous but to share it with the ratee. In such examples, the system may directly use the raw data in the first database 145 for reporting and analysis purposes.

[0093] The assessment may be holistic (considering all messages and numerical ratings), assessment type specific, or attribute specific. The assessment analysis software application includes computer algorithms and artificial intelligence that are programmed to perform the above operations. The assessment analysis software application is also configured to use the received assessment messages and numerical ratings to identify characteristics of the ratee. The assessment analysis software application uses data in the system 150 to make such determinations and assessments. In some embodiments, the learning behavior identification process may also take into account the user's given ratings. The assessment analysis software application 110 uses data in the system 150 such that the identity of the rater is kept anonymous. However, in some embodiments, the assessment analysis software application 110 may further consider data in a second database 155, if necessary.

[0094] Referring now to FIG. 9, a flow chart describes a general flow chart involved in identifying and classifying multi-dimensional data patterns, including attribute interrelationships (e.g., data relationships between pairs or groups of attributes, including additional dimensions of inter-rater relationships, such as instructor vs. self), according to an embodiment of the present invention, and may be executed by an assessment application and / or assessment processing software application in conjunction with a software engine and a reporting component. For example, once a classification or condition is detected, a system is implemented that stores a library of algorithms, each having a corresponding classification and associated stored data. Optionally, the software engine may be included within the analysis application 115 of FIG. 1. The software engine is configured to apply the library of algorithms (as a whole, individually, or in combination), such as by evaluating a numerical score for each attribute and detecting whether one or more of the algorithms are completed / fulfilled, e.g., detecting a classification. The algorithms may include many levels of complexity, such as being able to detect interrelationships between certain attributes. For example, whether the average of a first and a third attribute results in a higher numerical value than the score of a second attribute. Or, for example, whether the average of the first and third attributes results in a higher numerical value than the score of the second attribute, while the score for the second attribute is lower than both the fifth and fourth attributes, respectively. Complex analyses of many different patterns can be detected and classified from automated data analysis based on a corresponding library of information. The classification through the algorithm and the classification information stored can be used to display the detected classification or other automated assessment to the user through structured reports that are displayed to the user in the application. The reports can be displayed in conjunction with radar charts that provide an interactive radar display. The classification and reporting process provides the same data to (all) users within the enterprise once classifications are detected, and the classifications are provided only to the user's private space and selected instructors (the software limits access to the data).By "the same" data, this refers to the ability to report and display detected classifications to each user in the same way.

[0095] Referring again to FIG. 9, self-assessments and peer / mentor evaluations for a number of attributes are received (905). As discussed, the self-assessments and peer / mentor evaluations include numeric ratings and free-form text ratings for the attributes. As an example, the evaluations provide assessments related to five factors or attributes: (1) vision (as a measure of effectiveness in establishing goals and inspiring others toward them); (2) progress measurement (as a measure of effectiveness in identifying trends and maintaining perspective); (3) discipline in challenges (as a measure of consistency and discipline in tackling difficult tasks); (4) promptness and boldness (as a measure of promptness and boldness in action that takes into account an individual's ability to make tradeoffs); and (5) integrity (not lying, concealing, or cheating) (as a measure of how an individual communicates honestly with others and / or their own ability to pursue the truth). The assessment processing software application (e.g., the service application) is also configured, for example, to average the ratee's numerical assessment ratings per assessment criterion (in each assessment category, e.g., peer assessment and mentor assessment) and provide the assessment message and average numerical assessment rating in each category to the ratee via the assessment application software user interface (e.g., as shown in Figures 8A and 8B). The service application may access the assessment messages and ratings stored in the data cache and retrieval system upon request from the ratee and provide the assessment message and average rating to the ratee. The identity of the rater of each assessment message is kept anonymous, as discussed above.

[0096] The software engine then begins to analyze the assessment by generating one or more scores based on the received numerical ratings in the assessment (910). For example, without limitation, the software engine may calculate an overall attribute score received by the assessee (e.g., by adding up all the numerical values ​​for that attribute), an average attribute score received by the assessee (e.g., by averaging all the numerical values ​​for that attribute), an overall self, peer, and / or mentor score (e.g., by adding up the scores in the self-assessment, peer rating, and / or mentor rating), the difference in the self-assessment numerical value relative to that provided by peers and / or mentors (e.g., by averaging the numerical values ​​of an attribute provided by peers and / or mentors and determining the difference from that of the self-assessment value for that attribute), an average and / or overall score for a combination of one or more attributes and / or a comparison of such scores to other scores, or the like.

[0097] The software engine then performs a multi-level classification and analysis of the ratee's current behavior based on the calculated scores by detecting interrelationships between certain attributes (915). For example, the software engine may analyze the assessments and calculated scores to determine or detect certain patterns observed in the given assessment messages and numerical assessment ratings, and communicate a visual indicator to the ratee based on the determined pattern. In determining the pattern, the assessment analysis application 115 may parse the free-form assessment messages based on a large set of words and synonyms, and / or determine an additive score based on the free-form messages. The software may perform an analysis of each of the scores (e.g., calculated scores and / or additive scores), such as by traversing through point-by-point analysis to identify classifications from a library. In the case of self-assessment aimed at improving decision-making attributes, the software engine may use the calculated scores to classify the ratee's imbalances and relationship gaps with peers and / or mentors. For the overall score, the metric may classify the ratee's decision-making on a scale (e.g., average, above expectations, as expected, requires effort, etc.) for example, basis, meaning attachment, and action. The overall score may optionally consider only the numerical scores provided by the mentor rater for various attributes and classify them under various traits and corresponding scale levels. For imbalance, the determined metric may indicate or highlight biases that may distort the ratee's decision-making. For relationship gaps, the automated analysis may evaluate the gap between the ratee's self-understanding and that of his / her peers and mentors. The metric may indicate, for example, that the user is silent when he / she does not give any rating or does not receive enough ratings (e.g., below a threshold). This can also be understood as an automated interpretation engine that can be updated remotely, is private to the user, and is consistent for everyone in the enterprise.This can also be identified and classified by evaluating the scores underlying the indirect information (and / or other information in the assessment) about which the system has knowledge, based on a library of algorithms.

[0098] In various embodiments, the relationship gap between the ratee and his / her peers is determined by comparing the self-assessment numerical rating to that of the average numerical rating of the peers, and the relationship gap between the ratee and his / her mentor is determined by comparing the self-assessment for each criterion to that of the average numerical rating of the mentor. For example, the average score for each criterion of the peer ratings is compared to the corresponding self-assessment score for that criterion to determine the gap for that criterion. The gaps for each criterion are combined to determine an overall relationship gap in perception. Such an overall relationship gap in perception may be compared to one or more thresholds, for example, to determine whether the ratee's gap is high, medium, low, etc. with respect to his / her peers or mentors. EXAMPLES

[0099] Step 1: Example: Peer Rating A: 1, 2, 2, 2, 1 Peer Rating B: 1, 1, 1, 1, 1 Average colleague scorecard: 1, 1.5, 1.5, 1.5, 1 Step 2: Example: Self-rating: 2, 2, 2, 2, 2 Total gap with peers = 1 + 0.5 + 0.5 + 0.5 + 1 = 3.5 Step 3: Since the total peer gap is 3.5, this user would be classified under "high gap." High Gap: Total peer gap > 3 Medium Gap: Total peer gap > 1 and <= 3 Low Gap: Total peer gap <= 1

[0100] The gaps and relative shapes of pentagons 857, 858, and 859 may provide the assessee with an indication of his or her degree of agreement with peers and instructors and a corresponding explanation.

[0101] Ratee self-imbalance is determined by averaging the peer, mentor, and self-assessment numerical ratings for each assessment criterion and comparing them. For example, the following three sets of imbalances may be determined: ○Detailed strategist vs. theory-only professor ■Detail Strategist: The average combined score of the attributes of future vision and progress measurement is less than the average score of the attributes of discipline, speed / boldness, and conscientiousness in difficult situations. ■ Theory-only teaching: The mean scores for the attributes of future perspective and progress measurement exceed the mean scores for the attributes of discipline, promptness / boldness, and conscientiousness in difficult situations. o Recklessness: The average combined score of the attributes of future perspective and progress measurement is lower than the average score of the attribute of speed / boldness. No Signal: All attributes are <=1.5 by peers or instructors.

[0102] The overall score is determined using an average numeric rating of the assessments (including a combination of self, peer, and instructor assessments). For example, the overall score may be subdivided into categories such as base score (average combined score of attributes of discipline and conscientiousness in challenges), sensemaking (average combined score of attributes of vision and progress measurement), and action (average score of attributes of promptness / boldness). All three categories can be converted to expectations using thresholds such as below expectations (score <= 1.5), as expected (score > 1.5 and <= 2.5), above expectations (score > 2.5 and <= 3.5), and well above expectations (score > 3.5).

[0103] The generated information may be presented in the form of a report and / or a graphical representation that represents the user's assessment and shows the identified behavioral patterns (920).

[0104] For example, the assessment analysis application 115 may create a smart radar chart (e.g., 850 shown in FIG. 8A ) that includes a graphical user interface (GUI) that facilitates near real-time interactive visualization of multi-dimensional data. In the smart radar chart GUI, multiple dimensions of data are displayed radially from a central point. The radar chart is used, for example, when there are multiple assessment items for a certain measurement target. A circle (or another shape) is divided into sections each having an equal space according to the number of individual assessment items, the space to be measured is divided at equal intervals starting from the center of the circle in a concentric manner, dots are marked at corresponding positions according to the quantized scores of the individual assessment items, and the dots are connected between the assessment items to generate a line to represent the balance between the assessment items. The radar chart is advantageous in that several measurement targets are superimposed, so that the trends of the individual items can be easily recognized. For example, as shown in FIG. 8, radar chart 850 shows a ratee's self-rating superimposed with the average of one or more peer-rated numerical ratings and the average of one or more instructor-rated numerical ratings for five evaluation criteria (i.e., future perspective 851, progress measurement 852, discipline in difficult tasks 853, promptness / boldness 854, and conscientiousness 855) in the form of pentagons 857, 858, and 859, respectively. Radar charts may be created using methods now or hereafter known. For example, circles around a center point identify the average values ​​for the numerical ratings corresponding to each evaluation criterion, and color, shading, and / or other suitable visual indications are used to indicate whether a particular value is above or below the average for that dimension, as well as to distinguish between self-ratings, peer ratings, and instructor ratings. The visualization allows the user (e.g., assessee or instructor) to perceive exceptions in the data very quickly and easily, even if there are many visualized dimensions and / or the display area available for the visualization is small. Data points are plotted / displayed along the data dimension to which they correspond. The plotted data is normalized to a 50% circle, so that data points with normal values ​​would be plotted on a normal circle.The normal value may be, for example, an average, expected, or desired value for the data. Data having values ​​above the normal value (positive exceptions) would be plotted on one side of a normal circle, and data having values ​​below the normal value (negative exceptions) would be plotted on the other side of the normal circle.

[0105] A report is then generated by identifying the traits and characteristics or behavioral qualifications identified from the radar chart and / or the interpretation of the assessment (925). A specific example of a report on decision-making regarding the assessee from the radar chart 850 (801) is shown in FIG. 8B. As shown in FIG. 8B, section 861 includes the generated (two-way) radar chart, and section 862 includes the identified classifications and characteristics (e.g., relationship gaps in perception, including peer and mentor relationships, imbalance, and overall mentor score) along with a description of each identified trait. The user can see the radar chart and the interpretation / classification that the assessee has a high relationship gap with his peers / mentors, therefore he has extremely low self-awareness, is a detail tactician, scores high on recklessness (imbalance), has foundation, meaning giving, and action scores that exceed expectations, etc. These results are not subject to subjective analysis and can be used by the assessee for self-development (e.g., by selecting appropriate training modules).

[0106] The system is configured to operate in real time. When a ratee selects an assessor and applies for an assessment, an application invitation with a ratee identifier is transmitted to a server on which an assessment analysis software application is implemented, which processes the received information and applies for an assessment. Similarly, when a rater selects a send command and transmits an assessment message and a numerical assessment rating from an electronic device, the assessment message, the numerical assessment rating, and the identifier are transmitted to a server on which an assessment analysis software application is implemented, which processes the received information (e.g., the assessment message, the numerical assessment rating, and the identifier) ​​and provides them to the ratee. Individual users (ratee and / or assessor) see the application and / or assessment message and average assessment rating immediately, such as within 2, 3, 5, or 10 seconds or less (including a few milliseconds) after the ratee applies for an assessment and / or the assessor clicks on the send button. The processes performed by the reputation analysis software application refer to one or more of the previously described storing (with respect to the first database 145), replicating, removing, storing (with respect to the data caching and retrieval system 150), and updating steps. They may further include replicating and storing rater identifiers in the second database 155. The system can perform all these steps in real time in response to the reputation provider selecting a send command. This paragraph also applies to other commands and feeds selected by the user.

[0107] In some embodiments, interactive training links (not shown here) may be displayed on the user interfaces of Figures 8A and 8B for navigating to training or development tools (or suggestions for developing skills) for improving various gap areas and corresponding scores. Such training or development tools may be identified and / or developed by the software application based on identified behavior patterns, classification, or reporting information.

[0108] In some other embodiments, only the ratee and / or those invited by the ratee (e.g., mentors) are allowed to access the ratee's personal space or access the ratee's personal assessment information (data in their private space), such as viewing in the mentor's own interface. In some such embodiments, other personnel may be allowed to view aggregated and / or anonymized assessment data related to at least a threshold number of ratees (e.g., at least three or more people) in a manner such that the viewing personnel cannot associate the aggregated data with individual ratees. For example, personnel may be allowed to view the overall (or average) self-assessment and / or peer / mentor assessment scores of a group that includes at least a threshold number of ratees associated with an attribute as a radar chart. One method of anonymization is to aggregate data by averaging attributes for three or more people and removing personal identifiers such as names.

[0109] Real-time means that an electronic device, server, or other computer system performs an operation at a speed that appears to be instantaneous or near-instantaneous (from a human-perceived sense of time) relative to when the device receives input for the operation (e.g., a formed rating message, a selected numerical rating rating, etc.). The device, server, or computer system operates such that a user of the device, server, or computer system (e.g., an author or recipient of a rating message) perceives that an output (e.g., published and displayed rating messages and average numerical rating ratings) is produced instantaneously (e.g., without any understandable delay perceived by the user) when the device receives input or is selected to generate data or transmit received input or generated data. This also applies to sending a request from an electronic device, server, or other computer system to receiving a result at the electronic device, server, or other computer system in response to the request. A user of the device, server, or computer system perceives that a result is produced instantaneously when the user sends a request from the device, server, or computer system.

[0110] The assessment analysis software application and its server are configured as a service within an enterprise internal data network or domain (e.g., the server is an enterprise enterprise server). The assessment software application is configured to be installed and run on a mobile application, e.g., as an application installed using a mobile phone provider's application store. In some implementations, an enterprise may have employees download the application, which is supported by a vendor using a cloud service, e.g., that stores submitted assessments and provides the described functionality as a service to the enterprise (as opposed to, e.g., installing software on the enterprise's enterprise server).

[0111] The valuation analysis software application is configured in an arrangement in which the valuation software application is configured primarily as an interface for viewing and interacting with the valuation analysis software application implemented on the enterprise server. Preferably, processing or analytical capabilities are configured primarily or substantially only on the server on which the valuation analysis software application is implemented, and the valuation software application acts as an interactive viewer and data entry software client (enabling interaction with the server and the underlying volumes of data).

[0112] A schema is constructed having relationally linked data elements that are used by the reputation analysis software application to quickly assemble different messages, ratings, or responses based on specific circumstances. For example, a received request can be translated into a specific query to the service application regarding the received ratings of individual employees, and the data can be read from the readout system and second database and translated into a message (or messages) that is transmitted to the requesting reputation analysis software application.

[0113] The reputation analysis software application and gateway can be implemented using a cloud application service provider. For example, a cloud provider such as Amazon can be used and the associated applications, databases, and systems can be implemented on Amazon's cloud (e.g., simply a data center housing servers by the cloud provider that run applications for the cloud provider's customers, e.g., companies that want to run a reputation system for their company in the cloud, and the installation and launch of the software is provided as a service to those customers). Cloud-based software applications, servers, or other computer systems allow electronic devices to communicate with the application, server, or system without relying on an enterprise or internal network. A user of an electronic device can form a reputation message, select a reputation rating, and send both to the application, server, or system from a location different from or remote from the location of the building where the enterprise or internal network is implemented and / or where the individual is performing the activity. A user can use the reputation software application and communicate with the reputation analysis software application and other applications using local and / or telecommunication networks (e.g., GSM, CDMA, 4G, etc.). Optionally, a virtual container system may be provided for an enterprise to provide bundled software, libraries, and configuration files for executing embodiments of the systems and methods of the present invention in a manner isolated from other virtual machine instances / applications / data / containers to preserve the security and integrity of the information contained therein.For example, referring to FIG. 1, the application environment can be configured to be a running service on a cloud service provider and supporting infrastructure, such as AWS by Amazon, and in some cases, as an additional measure of security, individual application environments (sometimes referred to as dockers or virtual machines) are implemented for corresponding clients of the service (the assessment service is provided to and managed from multiple clients / enterprises). Otherwise, the application environment is configured to serve multiple enterprise / entity clients. In the situation where there is a specific instance for an enterprise, the application and data will be separate with the cloud infrastructure to run and support only that enterprise. The instance will not support other clients. The service provider of the assessment application will have a system configured to update the algorithms and corresponding information for customers via the cloud as new patterns or classifications are developed.

[0114] The system is configured to function on an individual's personal smartphone, allowing them to log into the evaluation software application from their smartphone based on the system verifying and authenticating that they are an employee of the business entity. This then provides a security token that authorizes the user to use the evaluation software application on their smartphone. Communications for login and subsequent application use are configured to use existing smartphone and available network capabilities, such as communicating over a public network such as the Internet (as opposed to being limited to communications within the business entity's private network, for example). The ability to create a trusted environment for the user is one of the features provided by embodiments of the present invention. Through this feature, the user knows that their communications are secure and that individuals on or with access to the system are not random people from the public. An example of an authorization process and system for establishing a trusted environment is illustratively described in the attached application.

[0115] The assessment software application may also be a tutoring software application. The tutoring software application provides employees with a platform for collaborative training and skill development with selected tutors. The tutoring software application allows the user to share his assessment, goals, and skill development strategies with people he trusts, such as tutors. By sharing, this allows the tutors to understand the individual's current assessment and provide correct advice and suggestions. The tutoring software application provides a secure and trusted environment.

[0116] The rating recipient's electronic device may notify the recipient when the recipient receives the rating, such as via a push notification in the electronic device's status bar, email, text message, or other form of notification.

[0117] Anonymity means that the identity of the assessor is protected from the assessment recipient. The assessment recipient is not able to identify the assessment provider from the assessment software application or the data provided to the assessment software application by the assessment processing software application. The assessment recipient is also not able to trace the assessment submission process to identify the assessment provider. The data provided to the assessment software application by the assessment processing software application does not contain any sender identifier or has an encrypted sender identifier.

[0118] The rater identifier may also include any other information that identifies a user or sender, other than an identifier, number, or code itself.

[0119] The term system refers to the overall system for submitting, collecting, and analyzing individual assessments, not to a system within that system (e.g., an assessment processing system), unless otherwise stated.

[0120] It should be appreciated that in the art, a distinction exists between feedback and evaluation: feedback solutions relate to providing feedback regarding an individual's specific actions or work products, while evaluation refers to an overall assessment of attributes, not specifically with respect to that individual's specific actions or work products.

[0121] The illustrative systems and methods described herein include embodiments that provide a trusted anonymity system. This can be important to accomplish because it allows users to "enter" the evaluation environment for a particular business, organization, or group while at the same time providing a secure way to maintain anonymity within the system. For example, the combination can provide anonymity while also maintaining a trusted environment. This gives users the comfort that other users are their coworkers and not the general public who may view evaluations or comment (or troll) system participants. This may involve checking the current employee roster before providing access to the evaluation software application or other applications described in this disclosure.

[0122] The electronic device may be a desktop computer, a laptop computer, a tablet computer, a cellular device, a mobile device, a smart phone, a personal digital assistant (PDA), or other similar computer system having a microprocessor and a memory.

[0123] A microprocessor may include an application specific integrated circuit (ASIC), a programmable logic array (PLA), a digital signal processor (DSP), a field programmable gate array (FPGA), or any other integrated circuit. A microprocessor may also include one or more of a CPU, an application processor, or any other applicable processor, such as a system-on-chip that combines one or more of a reduced instruction set computing (RISC) processor.

[0124] Memory may include volatile memory, non-volatile (non-transient) memory, or both. Volatile memory may include cache memory, semi-permanent memory such as RAM, and / or one or more types of memory used to temporarily store computer data and instructions. Non-volatile memory may include read-only memory (ROM) or other static storage devices for storing static information and instructions for a microprocessor. System applications described herein (e.g., evaluation software applications, evaluation processing software, etc.) are implemented as computer instructions that may be stored in memory and that are executable by or may be configured to perform the functions or steps described herein.

[0125] The electronic devices also include storage devices, which may include hard disk drives and / or removable storage drives, representing floppy disk drives, magnetic tape drives, optical disk drives, and the like.

[0126] The network may be a communications network using any suitable communications protocol, such as radio frequency systems such as Wi-Fi, 802.11, Bluetooth, 900 MHz, 1.4 GHz, and 5.6 GHz communications systems, infrared, GSM+EDGE, CDMA, quad-band, and other cellular protocols, VOIP, or any other suitable protocol. The network may also be established by using wires such as optical fibers or Ethernet cables. The network connection interface may be a hardware device configured to support the above communications protocols, such as an Integrated Services Digital Network (ISDN) card, a Local Area Network (LAN) card, an Ethernet card, a cable modem, a satellite modem, or a modem for providing a data communications connection to a corresponding type of telephone line.

[0127] A server refers to a computer system or communications-oriented computer that includes the computer components mentioned above and typically includes faster processors, larger memory, greater storage capabilities, and the general ability to sustain simultaneous data communications with multiple end users, client devices, or electronic devices.

[0128] A computer system may refer to any electronic device, server, or other computer within this application that includes a microprocessor and memory.

[0129] Although some disclosures refer only to software applications, it should be understood that they also apply to the underlying computer systems on which the software applications are implemented.

[0130] Corresponding method and computer readable medium embodiments will be understood from the above and the disclosure as a whole. Also, broader, narrower, or different combinations of the described features are envisioned, such as, for example, features may be removed or added in a broadening or narrowing manner.

[0131] From the above description, it should be understood that the functionality and features of the systems, devices, or methods of the embodiments of the present invention include generating and transmitting signals to perform actions.

[0132] It will be understood that variations, clarifications, or modifications are contemplated. Applications of the technology to other fields are also contemplated.

[0133] Exemplary systems, devices, and methods are described for illustrative purposes. Moreover, since numerous modifications and changes will be readily apparent to those skilled in the art, it is not desired to limit the invention to the precise structures set forth in this disclosure. Hence, any suitable modifications and equivalents falling within the scope of the present invention may be used.

[0134] Thus, for example, any sequence and / or time order of steps of various processes or methods (or device connections or sequences of operations) described herein is illustrative and should not be construed as limiting. Thus, while the steps of various processes or methods or connections or sequences of operations may be shown and described as being in a sequence or time order, it should be understood that they are not necessarily limited to being performed in any particular sequence or order. For example, steps in such processes or methods may generally be performed in a variety of different sequences and orders while still falling within the scope of the invention. Also, in some discussions, it will be apparent to one of ordinary skill in the art that a subsequent action, process, or feature is responsive to an earlier action, process, or feature.

[0135] It is also implicit, and should be understood, that any application or system illustratively described herein provides computer-implemented functionality that performs a process or process step automatically, unless the description explicitly describes user intervention or manual action.

[0136] The description mainly discusses the system or apparatus and corresponding method and non-transitory computer readable medium embodiments that will be understood from the above and the entire disclosure. The computer readable medium may be a permanent or semi-permanent memory such as a hard drive, a floppy drive, an optical disk, a flash memory, a ROM, an EPROM, an EEPROM, etc., as would be known to those skilled in the art. The computer readable medium stores computer instructions executable by a microprocessor, and the execution of the instructions causes the microprocessor to perform the steps or functions described in this disclosure. Also, broader, narrower, or different combinations of the described features are envisioned, such as features may be removed or added in a broader or narrower manner.

[0137] It should be understood that claims containing fewer limitations, broader claims, such as claims that do not require certain features or process steps in the appended claims or this specification, clarifications of claim elements, different combinations, and alternative implementations or different uses based on this specification are also contemplated by embodiments of the invention.

[0138] It is to be understood that combinations of the described features or steps are contemplated even if they are not directly described together or in the same context.

[0139] The terms or words used herein are intended for those skilled in the art, and the meaning of those terms or words will be understood from the terminology used in the art, or can be reasonably interpreted based on the plain English meaning of the words in conjunction with knowledge in the art, including, for example, the understanding of implied features that may involve multiple possibilities, but which would be understood as a reasonable or primary understanding or meaning to a person skilled in the art.

[0140] The software may be implemented as distinct modules or software applications, or may be integrated together into an overall application, such as one that includes a user interface and handles other features to provide functionality to a user.

[0141] It is intended that the specification and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.

Claims

1. 1. A computer-implemented method for providing a guided interactive self-assessment service, comprising: implementing an application on a server, said application configured to provide said services to employees of the enterprise; implementing the application to provide an interactive user interface to the employee, the interactive user interface adapted to provide an automated self-assessment process to an individual user; implementing the application to establish a private space that the individual user may configure to limit access to the user's private space by other employees, the application being configured to provide the individual user with access to the private space via a browser and configured to invite one or more other users to have access to the individual user's private space; enabling the individual users to use the application to perform an automated self-assessment, the assessment comprising a numerical rating on five attributes and a free-form text description; enabling the individual user to use the application to self-select other employees on the service to perform peer ratings of the individual user, the peer ratings comprising the numerical ratings on the five attributes and one or more additional free-form text descriptions; using said server to receive and store said numerical ratings and said free-form text descriptions for said five attributes; configuring the application to store a library of algorithms, each configured to detect a classification by detecting a pattern in the numerical ratings that satisfies one of the algorithms; detecting one or more of said classifications for an individual user using the received numerical ratings for the five attributes from the individual user's self-rating and a plurality of the peer ratings for that individual user; providing the ability for an individual user to view, within the individual user's private space, a structured report based on the numerical ratings from the self-assessment and peer ratings, the report comprising the detected classification for the individual user, using the graphical user interface, and in conjunction with providing the structured report, displaying an interactive radar chart displaying a combination of the numerical ratings of the self-assessment and other ratings for the individual user, each as a connected polygon shape; storing the structured report including the detected classifications of the individual users; configuring the application to allow the user to return to use the application at a later time to perform the self-assessments and peer assessments again and to allow the user to review historical data from previous reports within the private space; 4. A computer-implemented method comprising:

2. The method of claim 1 , comprising anonymizing the ratings such that they are not associated with an identity of the one or more colleagues.

3. The method of claim 2 , wherein the step of anonymizing comprises removing identifying information from ratings received as part of the step of storing the ratings.

4. The method of claim 1 , wherein the method includes implementing the application on a company's enterprise network.

5. The method of claim 1 , wherein the method includes providing a list of employees of the enterprise for a user to select from to request a peer evaluation.

6. The method of claim 1 , further comprising updating the algorithm at another later date.

7. A non-transitory computer readable medium having stored thereon one or more software applications that cause a computer system to execute a method for implementing a guided interactive self-assessment service, the method comprising: implementing an application on a server, said application configured to provide said services to employees of the enterprise; implementing the application to provide an interactive user interface to the employee, the interactive user interface adapted to provide an automated self-assessment process to an individual user; implementing the application to establish a private space that the individual user may configure to limit access to the user's private space by other employees, the application being configured to provide the individual user with access to the private space via a browser and configured to invite one or more other users to have access to the individual user's private space; enabling the individual users to use the application to perform an automated self-assessment, the assessment comprising a numerical rating on five attributes and a free-form text description; enabling the individual user to use the application to self-select other employees on the service to perform peer ratings of the individual user, the peer ratings comprising the numerical ratings on the five attributes and one or more additional free-form text descriptions; using said server to receive and store said numerical ratings and said free-form text descriptions for said five attributes; configuring the application to store a library of algorithms, each configured to detect a classification by detecting a pattern in the numerical ratings that satisfies one of the algorithms; detecting one or more of said classifications for an individual user using the received numerical ratings for the five attributes from the individual user's self-rating and a plurality of the peer ratings for that individual user; providing the ability for an individual user to view, within the individual user's private space, a structured report based on the numerical ratings from the self-assessment and peer ratings, the report comprising the detected classification for the individual user, using the graphical user interface, and in conjunction with providing the structured report, displaying an interactive radar chart displaying a combination of the numerical ratings of the self-assessment and other ratings for the individual user, each as a connected polygon shape; storing the structured report including the detected classifications of the individual users; configuring the application to allow the user to return to use the application at a later time to perform the self-assessments and peer assessments again and to allow the user to review historical data from previous reports within the private space; 16. A non-transitory computer readable medium comprising:

8. 8. The non-transitory computer-readable medium of claim 7, wherein the method includes anonymizing the ratings such that they are not associated with an identity of the one or more colleagues.

9. 10. The non-transitory computer-readable medium of claim 8, wherein the anonymizing step comprises removing identifying information from ratings received as part of the storing the ratings step.

10. The non-transitory computer-readable medium of claim 7 , wherein the method includes implementing the application on an enterprise network of a company.

11. 8. The non-transitory computer readable medium of claim 7, wherein the method includes providing a list of employees of the enterprise for a user to select from to request a peer evaluation.

12. The non-transitory computer readable medium of claim 7 , wherein the method includes updating the algorithm at another later date.

13. 1. A computer-implemented system for providing a guided interactive self-assessment service, comprising: one or more computers configured to provide said services using an application using computer readable instructions stored in a non-transitory computer memory, said application configured on a server to provide said services to employees of the enterprise; the application is configured to provide an interactive user interface to the employee, the interactive user interface adapted to provide an automated self-assessment process to an individual user; the application is configured to establish a private space that the individual user may configure to limit access to the user's private space by other employees, the application is configured to provide the individual user with access to the private space via a browser, and is configured to invite one or more other users to have access to the individual user's private space; the application is configured to enable the individual user to use the application to perform an automated self-assessment, the assessment comprising a numerical rating on five attributes and a free-form text description; the application is configured to enable the individual user to use the application to self-select other employees to conduct peer ratings of the individual user on the service, the peer ratings comprising the numerical ratings on the five attributes and one or more additional free-form text descriptions; the application is configured to receive and store, using the server, the numerical ratings and the free-form text descriptions for the five attributes; the application is configured to store a library of algorithms, each configured to detect a classification by detecting a pattern in the numerical ratings that satisfy one of the algorithms; the application is configured to detect one or more of the classifications for an individual user using the received numerical ratings for the five attributes from the individual user's self-rating and a plurality of the peer ratings for the individual user; the application is configured to provide the ability for an individual user to use the graphical user interface to view, within the individual user's private space, a structured report based on the numerical ratings from the self-assessment and peer ratings, the report comprising the detected classification for the individual user, and in conjunction with providing the structured report, display an interactive radar chart displaying a combination of the numerical ratings of the self-assessment and other ratings for the individual user as respective connected polygon shapes; the application is configured to store the structured report including the detected classifications of the individual users; The application is configured to allow the user to return to use the application at a later time to perform the self-assessment and peer assessment again, and to allow the user to review historical data from previous reports within the private space. Computer implemented systems.

14. The computer implemented system of claim 13 , wherein the application is configured to anonymize the ratings such that they are not associated with an identity of the one or more colleagues.

15. The computer implemented system of claim 14 , wherein the application is configured to anonymize by removing identifying information from ratings received as part of storing the ratings.

16. The computer implemented system of claim 13 , wherein the application is implemented on an enterprise network of the company.

17. 14. The computer implemented system of claim 13, wherein the application is configured to provide a list of employees of the enterprise for a user to select from to request a peer rating.

18. The computer implemented system of claim 13 , wherein the application is configured to update and apply the algorithm at another later date.

Images