Automated empathetic evaluation of job candidates

Abstract

To provide automated empathetic assessment of a candidate for a job.SOLUTION: Operations include extracting first information about a first set of features of a first candidate, from a document or profile information of the first candidate. Second information about a second set of features, corresponding to the first set of features, is extracted from one or more databases. The second set of features is associated with a population of candidates with at least one demographic parameter same as that of the first candidate. A third set of features is determined based on difference of corresponding features from the first set of features and the second set of features. A pre-trained neural network model is applied on the third set of features to determine a set of weights associated with the third set of features. An empathy score of the first candidate is determined based on the set of weights. The empathy score of the first candidate is rendered.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments discussed in this disclosure relate to automated empathetic assessment of job candidates. [Background technology]

[0002] Advances in the fields of machine learning and artificial intelligence (AI) have led to the development of numerous machine learning and AI-based recruiting tools. Examples of such tools include human resource chatbots for interview scheduling, tools for writing job descriptions, conversational chatbots for candidate screening, tools for resume / candidate screening, and tools for profiling potential candidates based on current employees. However, certain machine learning and AI-based recruiting tools may be biased in their resume / candidate screening. For example, certain tools may be biased based on a candidate's gender (or age, race, or social class). Because such tools can often lead to the selection of candidates with similar demographics, the use of such tools may also reduce diversity in the workplace. Therefore, a solution is needed to evaluate job candidates with minimal bias.

[0003] The subject matter claimed in this disclosure is not limited to embodiments that operate only in such environments or that eliminate any of the disadvantages. Rather, this background is provided only to illustrate one example of the scope within which some embodiments described in this disclosure may be practiced. Summary of the Invention

[0004] According to an aspect of the embodiment, a method may include extracting first information regarding a first set of features associated with a first candidate from at least one of documents or profile information associated with the first candidate. The operations may further include extracting second information regarding a second set of features corresponding to the first set of features from one or more databases. The second set of features may be associated with a population of candidates including at least one of the same set of demographic parameters as the first candidate. The operations may further include determining a third set of features based on differences between the first set of features associated with the first candidate and corresponding features from the second set of features associated with the population of candidates. The operations may further include applying a pre-trained neural network model to the determined third set of features. The neural network model may be pre-trained to determine an empathy score for each candidate in the set of candidates based on the first set of features associated with the candidate set. The operations may further include determining a set of weights associated with the third set of features based on the application of the pre-trained neural network model. The operations may further include determining an empathy score associated with the first candidate based on the determined set of weights. The operations may further include rendering the determined empathy score associated with the first candidate.

[0005] The object and advantages of the embodiments will be realized and achieved at least by the elements, features, and combinations particularly pointed out in the claims.

[0006] Both the foregoing general description and the following detailed description are provided by way of example, and not limitation, of the invention as claimed.

[0007] Example embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings. [Brief explanation of the drawings]

[0008] [Figure 1]FIG. 1 illustrates an example environment for empathetic evaluation of a job application. [Figure 2] FIG. 1 is a block diagram representing an exemplary electronic device for empathetic evaluation of job candidates. [Figure 3] FIG. 1 depicts an exemplary processing pipeline for automated empathic evaluation of job candidates. [Figure 4] FIG. 1 depicts a flowchart of an example method for empathetic evaluation of job candidates. [Figure 5] FIG. 1 illustrates a flowchart of an example method for extracting and verifying first information associated with a first candidate. [Figure 6] FIG. 10 depicts a flowchart of an example method for extraction of second information related to a candidate population. [Figure 7] FIG. 1 depicts a flowchart of an example method for empathy score determination based on counterfactual criteria. [Figure 8] FIG. 1 depicts a flowchart of an example method for training a neural network model based on each of a set of documents assigned an empathy score by a set of raters. [Figure 9] FIG. 10 illustrates an example scenario for determining an empathy score associated with a first candidate. [Figure 10] FIG. 10 depicts an example scenario of empathy scores that may be determined for two candidates. [Figure 11] FIG. 1 illustrates an example scenario of a job description and a job candidate's resume. DETAILED DESCRIPTION OF THE INVENTION

[0009] All figures depicted in the drawings are in accordance with at least one embodiment described in this disclosure.

[0010] Some embodiments described herein relate to methods and systems for automated empathic evaluation of job candidates. In the present disclosure, first information regarding a first set of features associated with a first candidate may be extracted. The first information may be extracted from at least one of a document (e.g., a resume) or profile information associated with the first candidate. Furthermore, second information regarding a second set of features that may correspond to the first set of features may be extracted from one or more databases (e.g., one or more knowledge databases). The second set of features may be associated with a candidate population that includes at least one of the same demographic parameters as the first candidate. Thereafter, a third set of features may be determined based on corresponding feature differences from the first set of features associated with the first candidate and the second set of features associated with the candidate population. Furthermore, a pre-trained neural network model may be applied to the determined third set of features. The neural network model may be pre-trained to determine an empathy score for each candidate in the candidate set based on the first set of features associated with the candidate set. Further, a set of weights associated with the third set of features may be determined based on application of the pre-trained neural network model. An empathy score associated with the first candidate may be determined based on the determined set of weights. Further, the determined empathy score associated with the first candidate may be rendered.

[0011] In accordance with one or more embodiments of the present disclosure, the technical field of automated evaluation of job candidates may be improved by configuring a computing system to empathetically score job candidates with minimal bias. The computing system may determine a difference between a first set of characteristics of the candidate (e.g., but not limited to, educational background, certifications, languages ​​known, skill set) and corresponding characteristics in a second set of characteristics of a population of candidates belonging to a demographic background similar to the candidate. The computing system may determine an empathy score associated with the candidate based on the difference between the first set of characteristics associated with the candidate and the second set of characteristics associated with the population of candidates. The difference may indicate a greater skill acquisition ability and diligence of the candidate compared to other candidates with similar demographic backgrounds. The greater the difference, the higher the empathy score associated with the candidate, and the greater the likelihood of selection of the candidate for the job. The computing system may thereby include empathy as one of the factors for the automated evaluation of a job candidate based on the difference between the candidate's qualifications or skill set and that of a population of candidates with similar demographic backgrounds, compared to other conventional systems that may not consider such empathy in the evaluation of job candidates. The disclosed system may be advantageous in that the candidate's inherent skill acquisition ability and diligence, which may be useful for most positions, may be taken into account during the automated evaluation of the candidate based on the empathy score determined for the candidate. Also, based on the candidate's empathy score, the candidate may be given equal employment opportunities compared to other candidates with similar qualifications / skill sets but different demographic backgrounds. Thus, the use of the computing system may also help increase workforce diversity within an organization.

[0012] The system may be configured to extract first information regarding a set of first characteristics associated with the first candidate from at least one of documents or profile information associated with the first candidate. The document associated with the first candidate may be, for example, a resume or curriculum vitae of the first candidate. The profile information associated with the first candidate may include information associated with at least one of, but not limited to, a job cover letter, a set of recommendations, a source code repository, a project page, a professional networking webpage, or a social networking webpage associated with the first candidate. The set of first characteristics may include, but is not limited to, at least one of, a set of educational degrees, a set of certifications, years of work experience, location, a set of languages ​​studied or known, a disability, a set of skills, age, gender, or per capita income associated with the first candidate. Extraction of the first information is further described, for example, in Figures 3, 4, and 5.

[0013] The system may be configured to extract second information regarding a second set of features corresponding to the first set of features from one or more databases (e.g., one or more knowledge databases). The second set of features may be associated with a candidate population that includes at least one of the same set of demographic parameters as the first candidate demographic parameters. Examples of the set of demographic parameters may include, but are not limited to, location, household income, type of institution studied, educational background, certifications, skill set, family background, available resources, and languages ​​known or studied. The second set of features may include, but are not limited to, at least one of the following: average per capita income, set of educational degrees, set of certifications, number of job applications, number of institutions, number of graduate students, number of secured jobs, or set of languages ​​known or studied, associated with the candidate population. Extraction of the second information is further described, for example, in Figures 3, 4, and 5.

[0014] The system may be configured to determine a third set of features based on differences between the first set of features associated with the first candidate and corresponding features from a second set of features associated with the candidate population. The system may further be configured to apply a pre-trained neural network model to the determined third set of features. The neural network model may be pre-trained to determine an empathy score for each candidate in the set of candidates based on the first set of features associated with the candidate set. The system may be configured to determine a set of weights associated with the third set of features based on the application of the pre-trained neural network model. The system may further be configured to determine an empathy score associated with the first candidate based on the determined set of weights. The system may be configured to render the determined empathy score associated with the first candidate. Automatic empathy-based evaluation of job candidates is further described, for example, in FIGS. 3 and 4.

[0015] The system may be configured to apply a regularization constraint to the pre-trained neural network model based on a criterion associated with one or more protected features from the first feature set associated with the first candidate. The determination of the empathy score associated with the first candidate may be further based on applying the regularization constraint to the pre-trained neural network model. The criterion may correspond to counterfactual criteria (e.g., counterfactual fairness criteria) that a first likelihood of acceptance of a first candidate having a first value of a first protected feature of the one or more protected features is the same as a second likelihood of acceptance of a second candidate having a second value of the first protected feature, based on a determination that other values ​​of the first feature set are the same. Examples of the one or more protected (or sensitive) features may include, but are not limited to, at least one of age, gender, or physical disability associated with the first candidate. The application of the regularization constraint is further described, for example, in FIGS. 3 and 7.

[0016] The system may be further configured to extract a first set of named entities from at least one of documents or profile information associated with the first candidate, and further to filter the extracted first set of named entities to extract a second set of named entities. The system may further be configured to extract a first set of features associated with the first candidate based on excluding the extracted second set of named entities from the extracted first set of named entities. The system may be configured to verify the authenticity of first information associated with at least one of documents, profile information, or the first set of named entities associated with the first candidate based on one or more public information databases. The determination of an empathy score associated with the first candidate may further be based on the authenticity verification. The extraction of the first set of features and the authenticity verification of the first information are further described, for example, in FIGS. 3 and 5.

[0017] The system may be configured to send one or more queries to one or more databases (e.g., one or more knowledge databases). The one or more queries may be formed based on one or more combinations of a first set of features associated with the first candidates. Further, the system may be configured to extract second information related to a second set of features corresponding to the first set of features based on the sent one or more queries. Extraction of the second information based on sending one or more queries is further described, for example, in FIGS. 3 and 6.

[0018] The system may be configured to receive sets of empathy scores for sets of documents in a training dataset for a neural network model based on user input from one or more raters. The system may further be configured to calculate pairwise rankings among the received sets of empathy scores for the sets of documents. The system may be configured to determine an optimal empathy score ranking for each of the sets of documents based on minimizing the Kendall Tau distance across the calculated pairwise rankings. The system may further be configured to train a neural network model based on the determined optimal empathy score ranking for each of the sets of documents in the training dataset. Training the neural network model is further described, for example, in FIG. 8.

[0019] Typically, conventional systems may not incorporate empathy in evaluating job candidates. Conventional systems may use only information related to the job description, the candidate's educational background, and skill set to evaluate the candidate's job application, without considering the candidate's background factors, such as the candidate's family background, annual household income, and current / past resources available to the candidate. Furthermore, conventional systems may be biased with respect to certain factors, such as (but not limited to) the candidate's educational institution, the candidate's gender, and / or the candidate's age. In addition, conventional systems may select candidates with similar backgrounds for the job. The use of such conventional systems for recruiting may reduce the diversity of an organization's workforce. The disclosed system, on the other hand, may determine differences between a first set of characteristics (e.g., educational background, certifications, languages ​​known, skill set) of the candidate and corresponding characteristics in a second set of characteristics of a population of candidates belonging to a demographic background similar to that of the candidate. An empathy score associated with the candidate may then be determined based on the differences between the first set of characteristics associated with the candidate and the second set of characteristics associated with the population of candidates. The difference may indicate a candidate's skill acquisition ability and diligence compared to other candidates with similar demographic backgrounds. The greater the difference, the higher the empathy score associated with the candidate, increasing the likelihood of selection of the candidate for the job. The disclosed system may thereby include empathy as one of the factors for the automated evaluation of a job candidate based on the difference between the candidate's qualifications or skill set and that of a population of candidates with similar demographic backgrounds, compared to other conventional systems that may not consider such empathy in evaluating job candidates. The disclosed system may be advantageous in that a candidate's inherent skill acquisition ability and diligence, which may be useful for most positions, may be taken into account during the automated evaluation of the candidate based on the empathy score determined for the candidate. Also, based on the candidate's empathy score, the candidate may be given equal employment opportunities compared to other candidates with similar qualifications / skill sets but different demographic backgrounds. Thus, use of the disclosed system may also help increase workforce diversity within an organization.Furthermore, based on the incorporation of counterfactual fairness criteria as a regularization constraint, the disclosed system may be less biased with respect to various protected or sensitive characteristics (e.g., age, disability, and gender) compared to conventional systems.

[0020] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0021] FIG. 1 is a diagram illustrating an example environment for empathetic evaluation of job applications, arranged in accordance with at least one embodiment described in this disclosure. Referring to FIG. 1, environment 100 is shown. Environment 100 may include an electronic device 102, a candidate information database (DB) 104, one or more knowledge DBs 106, one or more public information DBs 108, a user end device 110, and a communication network 112. Electronic device 102, candidate information DB 104, one or more knowledge DBs 106, one or more public information DBs 108, and user end device 110 may be communicatively coupled to each other via communication network 112. One or more knowledge DBs 106 may include a first DB 106A, a second DB 106B, ..., and an NDB 106N. Furthermore, one or more public information DBs 108 may include a first DB 108A, a second DB 108B, ..., and an NDB 108N. FIG. 1 also shows a user 114 who may be associated with or operating the electronic device 102 or user end device 110. Also shown are first candidate information 120A associated with a first candidate, second candidate information 120B associated with a second candidate, and Nth candidate information 120N associated with an Nth candidate. The first candidate information 120A may include a first document 116A and first profile information 118A. The second candidate information 120B may include a second document 116B and second profile information 118B. Similarly, the Nth candidate information 120N may include an Nth document 116N and Nth profile information 118N. The first candidate information 120A, the second candidate information 120B, and the Nth candidate information 120N may be stored in the candidate information DB 104.

[0022] The electronic device 102 may include appropriate logic, circuitry, interfaces, and / or code that may be configured to empathically evaluate a candidate's job application based on determining an empathy score associated with the candidate. The electronic device 102 may retrieve candidate information for a candidate from the candidate information DB 104 for determining an empathy score associated with the candidate. For example, the electronic device 102 may retrieve first candidate information 120A associated with a first candidate from the candidate information DB 104. The first candidate information 120A may include a first document 116A and first profile information 118A. The first document 116A may include, for example, a resume or curriculum vitae associated with the first candidate. The first profile information 118A associated with the first candidate may include at least one of, but not limited to, a job cover letter, a set of recommendations, a source code repository, a project page, a professional networking webpage, or a social networking webpage associated with the first candidate. The electronic device 102 may be configured to extract first information regarding a first set of characteristics associated with the first candidate from at least one of, but not limited to, a document (e.g., first document 116A) or profile information (e.g., first profile information 118A) associated with the first candidate. The first set of characteristics may include, but is not limited to, at least one of, a set of educational degrees, a set of certifications, years of work experience, location, a set of languages ​​studied or known, a physical disability, a set of skills, age, gender, or per capita income associated with the first candidate. Extraction of the first information is further described, for example, in FIGS. 3, 4, and 5.

[0023] The electronic device 102 may be configured to extract second information related to a second set of features (i.e., corresponding to the first set of features) from one or more databases (e.g., one or more knowledge DBs 106). The second set of features may be associated with a population of candidates having at least one of the same set of demographic parameters as the demographic parameters of the first candidate. Examples of the set of demographic parameters may include, but are not limited to, household income, location, type of institution studied, educational background, certifications, skill sets, family background, available resources, and languages ​​known or studied. The second set of features may include, but are not limited to, at least one of the following: average per capita income, set of educational degrees, set of certifications, number of job applications, number of institutions, number of graduate students, number of secured jobs, or set of languages ​​known or studied, associated with the population of candidates. Extraction of the second information is further described, for example, in FIGS. 3, 4, and 5.

[0024] The electronic device 102 may be configured to determine a third set of features based on a difference between the first set of features associated with the first candidate and corresponding features from a second set of features associated with the candidate population. The electronic device 102 may further be configured to apply a pre-trained neural network model to the determined third set of features. The neural network model may be pre-trained to determine an empathy score for each candidate in the set of candidates based on the first set of features associated with the candidate set. The electronic device 102 may be configured to determine a set of weights associated with the third set of features based on the application of the pre-trained neural network model. The electronic device 102 may further be configured to determine an empathy score associated with the first candidate based on the determined set of weights. The electronic device 102 may be configured to render the determined empathy score associated with the first candidate. Empathy-based evaluation of job candidates is further described, for example, in FIGS. 3 and 4.

[0025] Examples of electronic device 102 may include, but are not limited to, a recruitment engine or machine, a mobile device, a desktop computer, a laptop, a computer workstation, a computing device, a mainframe machine, a server such as a cloud server, and a group of servers. In one or more embodiments, electronic device 102 may include a user-end terminal device and a server communicatively coupled to the user-end terminal device. Electronic device 102 may be implemented using hardware including a processor, a microprocessor (e.g., for performing or controlling one or more operations), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other cases, electronic device 102 may be implemented using a combination of hardware and software.

[0026] The candidate information DB 104 may include appropriate logic, interfaces, and / or code that may be configured to store candidate information, including documents and profile information associated with one or more candidates. The candidate information DB 104 may be a relational or non-relational database. In some cases, the candidate information DB 104 may be stored on a server, such as a cloud server, or may be cached and stored on the electronic device 102. The server of the candidate information DB 104 may be configured to receive a request for at least one of documents (e.g., first document 116A) and profile information (e.g., first profile information 118A) associated with a candidate (e.g., first candidate) from the electronic device 102 via the communications network 112. In response, the server of the candidate information DB 104 may be configured to retrieve and provide the requested documents and profile information to the electronic device 102 via the communications network 112 based on the received request. In some embodiments, the candidate information DB 104 may include multiple servers stored in different locations. Additionally or alternatively, the candidate information DB 104 may be implemented using hardware, including a processor, a microprocessor (e.g., for performing or controlling one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other cases, the candidate information DB 104 may be implemented using a combination of hardware and software.

[0027] One or more knowledge databases (DBs) 106 may include appropriate logic, interfaces, and / or code that may be configured to store demographic information and candidate information related to a population of candidates within a region. The one or more knowledge DBs 106 may be relational or non-relational databases. In some cases, the one or more knowledge DBs 106 may be stored on a server, such as a cloud server, or may be cached and stored on the electronic device 102. The one or more knowledge DB 106 servers may be configured to receive one or more queries from the electronic device 102 via the communications network 112 based on one or more combinations of a first set of features associated with a first candidate. In response, the one or more knowledge DB 106 servers may be configured to retrieve and provide second information related to a second set of features corresponding to the first set of features to the electronic device 102 via the communications network 112 based on the received request. In some embodiments, the one or more knowledge DBs 106 may include multiple servers stored in different locations. Additionally or alternatively, one or more knowledge DBs 106 may be implemented using hardware, including a processor, a microprocessor (e.g., for performing or controlling one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other cases, one or more knowledge DBs 106 may be implemented using a combination of hardware and software.

[0028] The one or more public information databases (DBs) 108 may include appropriate logic, interfaces, and / or code that may be configured to store public records (e.g., school records, university or college records, certification records, training records, performance records, and / or work experience records, but are not limited to) associated with one or more candidates. The one or more public information DBs 108 may be relational or non-relational databases. Also, in some cases, the one or more public information DBs 108 may be stored on a server, such as a cloud server, or may be cached and stored on the electronic device 102. The server of the one or more public information DBs 108 may be configured to receive a request for verification of the authenticity of first information associated with at least one of a document (e.g., first document 116A), profile information (e.g., first profile information 118A), or first set of named entities associated with the first candidate. The request may be received from the electronic device 102 via the communication network 112. In response, the one or more servers of the public information DB 108 may be configured to verify the authenticity of the first information to the electronic device 102 via the communication network 112 based on the received request. In some embodiments, the one or more servers of the public information DB 108 may include multiple servers stored in different locations. Additionally or alternatively, the one or more servers of the public information DB 108 may be implemented using hardware including a processor, a microprocessor (e.g., for performing or controlling one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other cases, the one or more servers of the public information DB 108 may be implemented using a combination of hardware and software.

[0029] The user end device 110 may include appropriate logic, circuitry, interfaces, and code that may be configured to generate or receive candidate information (e.g., first candidate information 120A for the first document 116A and first profile information 118A) for a candidate (e.g., a first candidate). For example, the user end device 110 may include web browser software or email software, through which the user end device 110 may receive the candidate information. Additionally or alternatively, the user end device 110 may include word or text processing software, through which the candidate information (e.g., resume and / or profile information) for the candidate may be generated based on user input from the user 114 or from the candidate themselves. The user end device 110 may upload the generated or received candidate information related to the candidate (e.g., the first candidate) to the electronic device 102 for extraction of first information related to the candidate. Additionally, the user end device 110 may upload the generated or received candidate information to the candidate information DB 104 for storage. The user end device 110 may be further configured to receive an empathy score associated with the candidate (e.g., a first candidate) from the electronic device 102. The user end device 110 may render the received empathy score associated with the candidate on a display screen of the user end device 110 for the user 114. In some embodiments, the user end device 110 may receive a request from the user 114 (e.g., a recruiter or any employee of the organization) to determine an empathy score for the candidate (e.g., a first candidate). The user end device 110 may further send such a request to the electronic device 102 for determination of an empathy score for the candidate. Examples of user end devices 110 may include, but are not limited to, a mobile device, a desktop computer, a laptop, a computer workstation, a computing device, a mainframe machine, a server such as a cloud server, and a group of servers.In FIG. 1, the user end device 110 is separate from the electronic device 102, but in some embodiments, the user end device 110 may be incorporated into the electronic device 102 without departing from the scope of the present disclosure.

[0030] The communication network 112 may include a communication medium through which the electronic device 102 may communicate with various databases (e.g., the candidate information DB 104, one or more knowledge DBs 106, and one or more public information DBs 108) and the user end devices 110. Examples of the communication network 112 may include, but are not limited to, the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). The various devices in the environment 100 may be configured to connect to the communication network 112 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communications, wireless access points (APs), device-to-device communications, cellular communications protocols, and / or Bluetooth (BT) communications protocols, or combinations thereof.

[0031] Modifications, additions, or omissions may be made to FIG. 1 without departing from the scope of the present disclosure. For example, environment 100 may include more or fewer elements than those shown and described in this disclosure. For example, in some embodiments, environment 100 may include electronic device 102 but not various databases (e.g., candidate information DB 104, one or more knowledge DBs 106, and one or more public information DBs 108) and user end devices 110. Moreover, in some embodiments, the respective functionality of various databases (e.g., candidate information DB 104, one or more knowledge DBs 106, and one or more public information DBs 108) and user end devices 110 may be incorporated into electronic device 102 without departing from the scope of the present disclosure.

[0032] Figure 2 is a block diagram representing an example electronic device for empathic evaluation of job candidates, arranged in accordance with at least one embodiment described in the present disclosure. Figure 2 will be described in conjunction with elements from Figure 1. With reference to Figure 2, a block diagram 200 of a system 202 including an electronic device 102 is shown. The electronic device 102 may include a processor 204, a memory 206, persistent data storage 208, input / output (I / O) devices 210, a display screen 212, a network interface 214, and a neural network model 216.

[0033] The processor 204 may include appropriate logic, circuitry, and / or interfaces that may be configured to execute program instructions associated with various operations performed by the electronic device 102. For example, some of the operations may include extracting first information, extracting second information, determining a third set of features, and applying a pre-trained neural network model (e.g., neural network model 216) to the determined third set of features. The operations may further include determining a set of weights associated with the third set of features, determining an empathy score associated with the first candidate, and rendering the determined empathy score. The processor 204 may include any suitable special-purpose or general-purpose computer, computing entity, or processing device, including various computer hardware or software modules, and may be configured to execute instructions stored on any applicable computer-readable storage medium. For example, processor 204 may include a microprocessor, a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any other digital or analog circuit configured to interpret and / or execute program instructions and / or process data.

[0034] 2 as a single processor, processor 204 may include any number of processors configured to individually or collectively perform or direct the performance of any number of the operations of electronic device 102 described in this disclosure. Further, one or more of the processors may reside in one or more different electronic devices, such as different servers. In some embodiments, processor 204 may be configured to interpret and / or execute program instructions and / or process stored data stored in memory 206 and / or persistent data storage 208. In some embodiments, processor 204 may fetch program instructions from persistent data storage 208 and load the program instructions into memory 206. After the program instructions are loaded into memory 206, processor 204 may execute the program instructions. Some examples of processor 204 may be a graphics processing unit (GPU), a central processing unit (CPU), a reduced instruction set computer (RISC) processor, an ASIC processor, a complex instruction set computer (CISC) processor, a co-processor, and / or combinations thereof.

[0035] Memory 206 may include appropriate logic, circuits, interfaces, and / or code that may be configured to store program instructions executable by processor 204. In particular embodiments, memory 206 may be configured to store an operating system and associated application-specific information. Memory 206 may include computer-readable storage media for carrying or storing computer-executable instructions or data structures. Such computer-readable storage media may include any available media that can be accessed by a general-purpose or special-purpose computer, such as processor 204. By way of example, and not limitation, such computer-readable storage media may include tangible or non-transitory computer-readable storage media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices), or any other storage medium that can be used to carry or store specific program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable media. Computer-executable instructions may include, for example, instructions and data configured to cause processor 204 to perform a certain operation or group of operations related to electronic device 102.

[0036] Persistent data storage 208 may include appropriate logic, circuits, interfaces, and / or code that may be configured to store program instructions executable by processor 204, an operating system, and / or application-specific information, such as logs and application-specific databases. Persistent data storage 208 may include computer-readable storage media for carrying or storing computer-executable instructions or data structures. Such computer-readable storage media may include any available media that can be accessed by a general-purpose or special-purpose computer, such as processor 204.

[0037] By way of example, and not limitation, such computer-readable storage media may include tangible or non-transitory computer-readable storage media, including compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices (e.g., hard disk drives (HDD)), flash memory devices (e.g., solid-state drives (SSD), Secure Digital (SD) cards, other solid-state memory devices), or any other storage medium that can be used to carry or store specific program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable media. Computer-executable instructions may include, for example, instructions and data configured to cause processor 204 to perform a certain operation or group of operations associated with electronic device 102.

[0038] In some embodiments, either memory 206 or persistent data storage 208, or a combination thereof, may store candidate information (e.g., first candidate information 120A including first document 116A and first profile information 118A) for a candidate (e.g., first candidate) retrieved from candidate information DB 104. Either memory 206 or persistent data storage 208, or a combination thereof, may further store information related to the extracted first information, the extracted second information, the determined set of third features, the determined set of weights, and the determined empathy score associated with the candidate (e.g., first candidate).

[0039] The neural network model 216 may be a computational network or system of artificial neurons arranged as nodes in multiple layers. The multiple layers of the neural network model 216 may include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers may include one or more nodes (or artificial neurons, represented by circles). The outputs of all nodes in the input layer may be connected to at least one node in the hidden layer. Similarly, the inputs of each hidden layer may be connected to the outputs of at least one node in another layer of the neural network model 216. The outputs of each hidden layer may be connected to the inputs of at least one node in another layer of the neural network model 216. A node in a final layer may receive inputs from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer may be determined from hyperparameters of the neural network model 216. Such hyperparameters may be set before or during training the neural network model 216 on a training dataset.

[0040] Each node of the neural network model 216 may correspond to a mathematical formula (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of the neural network model 216. The set of parameters may include, for example, weight parameters, regularization parameters, etc. Each node may use the mathematical formula to calculate an output based on one or more inputs from nodes in other layers (e.g., previous layers) of the neural network model 216. All or some of the nodes of the neural network model 216 may correspond to the same or different mathematical formulas.

[0041] In training the neural network model 216, one or more parameters of each node of the neural network model 216 may be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on the loss function of the neural network model 216. The above process may be repeated for the same or different inputs until a minimum of the loss function can be achieved and the training error can be minimized. Several training methods, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, meta-heuristics, etc., are known in the art.

[0042] The neural network model 216 may include electronic data, such as, for example, a software program, software program code, library, application, script, or other logic or instructions for execution by a processing device, such as the processor 204. The neural network model 216 may include code and routines configured to enable a computing device, including the processor 204, to perform one or more tasks, such as determining an empathy score for a candidate based on a set of features associated with the candidate and / or determining a set of weights associated with a set of features associated with the candidate. Additionally or alternatively, the neural network model 216 may be implemented using hardware, including a processor, a microprocessor (e.g., for performing or controlling one or more operations), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural network model 216 may be implemented using a combination of hardware and software.

[0043] Examples of the neural network model 216 may include, but are not limited to, a deep neural network (DNN), a convolutional neural network (CNN), an artificial neural network (ANN), a recurrent neural network (RNN), a CNN-recurrent neural network (CNN-RNN), a long short-term memory (LSTM) network-based RNN, an LSTM+ANN, a connectionist temporal classification (CTC)-based RNN, a fully connected neural network, a deep Bayesian neural network, and / or a combination of such networks. In some embodiments, the neural network model 216 may include numerical computation techniques using data flow graphs. In certain embodiments, the neural network model 216 may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0044] The I / O device 210 may include appropriate logic, circuitry, interfaces, and / or code that may be configured to receive user input. For example, the I / O device 210 may receive user input to retrieve candidate information (e.g., first candidate information including the first document 116A and the first profile information 118A) associated with a candidate (e.g., the first candidate). In other examples, the I / O device 210 may receive user input to create a new document or profile information, to edit an existing document (e.g., the retrieved first document 116A) or existing profile information (e.g., the retrieved first profile information 118A), and / or to store the created / edited document or profile information. The I / O device 210 may further receive user input, which may include instructions for determining an empathy score associated with the candidate (e.g., the first candidate). The I / O device 210 may further be configured to provide an output in response to the user input. For example, the I / O device 210 may render an empathy score (e.g., which may be determined by the electronic device 102) associated with the first candidate on the display screen 212. The I / O device 210 may include various input and output devices that may be configured to communicate with the processor 204 and other components, such as the network interface 214. Examples of input devices may include, but are not limited to, a touchscreen, a keyboard, a mouse, a joystick, and / or a microphone. Examples of output devices may include, but are not limited to, a display (e.g., the display screen 212) and a speaker.

[0045] The display screen 212 may have appropriate logic, circuitry, interfaces, and / or code that may be configured to display the empathy score associated with a candidate (e.g., the first candidate). The display screen 212 may be configured to receive user input from the user 114. In such a case, the display screen 212 may be a touch screen that receives the user input. The display screen 212 may be implemented through several known technologies, such as, but not limited to, liquid crystal display (LCD) display, light emitting diode (LED) display, plasma display, and / or organic LED (OLED) display technology, and / or other display technologies.

[0046] The network interface 214 may comprise appropriate logic, circuitry, interfaces, and / or code that may be configured to establish communications between the electronic device 102, various databases (such as the candidate information DB 104, the one or more knowledge DBs 106, and the one or more public information DBs 108), and the user end device 110 over the communications network 112. The network interface 214 may be implemented using various known technologies to facilitate wired or wireless communications of the electronic device 102 over the communications network 112. The network interface 214 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0047] Modifications, additions, or deletions may be made to the example electronic device 102 without departing from the scope of the present disclosure. For example, in some embodiments, the example electronic device 102 may include any number of other components that may not be explicitly shown or described for the sake of brevity.

[0048] Figure 3 is a diagram depicting an exemplary processing pipeline for automated empathic evaluation of job candidates according to an embodiment of the present disclosure. Figure 3 will be described in conjunction with elements from Figures 1 and 2. Referring to Figure 3, a processing pipeline 300 of operations 302 through 314 for automated empathic evaluation of job candidates is shown.

[0049] At 302, data may be acquired. In an embodiment, the processor 204 may be configured to acquire data related to the candidate. The acquired data may include candidate information related to the candidate for whom an empathy score is to be determined. For example, the processor 204 may acquire first candidate information 120A related to a first candidate. The first candidate information 120A may include a first document 116A, such as a resume related to the first candidate. The first candidate information 120A may further include first profile information 118A, such as, but not limited to, information related to at least one of a job cover letter, a set of recommendations, a source code repository, a project-related page, one or more professional networking web pages, or a social networking web page related to the first candidate. The processor 204 may extract the first candidate information 120A from the candidate information DB 104 and store the extracted first candidate information 120A in the memory 206 and / or persistent data storage 208. Alternatively, the processor 204 may obtain the first candidate information 120A based on a user input.

[0050] At 304, a set of named entities may be recognized. In an embodiment, the processor 204 may be configured to recognize a set of named entities from the retrieved data (the data retrieved in act 302). Examples of techniques or tools used to recognize a set of named entities from the retrieved data may include, but are not limited to, Nlp.js, spark.js, Stanford NLPcore, or Tensorflow sequence tagging. Examples of a set of named entities that may be recognized from the first candidate information 120A may include, but are not limited to, education, certifications, location, marital status, languages ​​known / studied, disability, skill set, age, gender, and household / capita income associated with the first candidate.

[0051] At 306, the recognized named entities may be filtered. In an embodiment, the processor 204 may be configured to filter the set of recognized named entities obtained at (operation 304). Filtering the set of recognized named entities may include excluding certain named entities from the set of recognized named entities based on predetermined rules or user input. Examples of techniques or tools that may be used to filter the recognized named entities may include, but are not limited to, Stanford NLPcore or Tensorflow sequence tagging. Examples of named entities that may be obtained after filtering may include, but are not limited to, education, certifications, location, marital status, languages ​​known / studied, disability, skill set, age, gender, and household / capita income associated with the first candidate. In the above example, the marital status of the first candidate may be excluded from the set of recognized named entities if it may not be required for determining the empathy score associated with the first candidate. In an embodiment, the processor 204 may extract first information regarding a first set of features associated with the first candidate information 120A based on filtering the set of recognized named entities. Examples of the first set of features may include at least one of a set of educational degrees, a set of certifications, years of work experience, location, a set of languages ​​studied or known, a physical disability, a set of skills, age, gender, or per capita income associated with the first candidate. Extracting the first information regarding the first set of features associated with the first candidate is further described, for example, in FIG. 5.

[0052] At 308, one or more databases (e.g., one or more knowledge DBs 106) may be queried. In an embodiment, the processor 204 may be configured to send one or more queries to one or more databases (i.e., one or more knowledge DBs 106) based on one or more combinations of the first set of features associated with the first candidate. Based on the sent one or more queries, the processor 204 may be configured to extract second information related to a second set of features. The second set of features may be associated with a population of candidates having at least one of the set of demographic parameters that is the same as the demographic parameters of the first candidate. The candidate population may be candidates different from the first candidate for whom an empathy score is to be determined. The candidate population may be selected to be likely to have one or more demographic parameters similar to that of the first candidate. Examples of the set of demographic parameters may include, but are not limited to, household income, location, type of institution studied, educational background, certifications, skill set, family background, available resources, and languages ​​known or studied. Examples of the set of second features may include, but are not limited to, at least one of the following: average per capita income, set of educational degrees, set of certifications, number of job applications, number of graduate students, number of institutions, number of jobs secured, or set of languages ​​studied or known, associated with the candidate population. Extraction of second information related to the set of second features is further described, for example, in FIG. 6.

[0053] At 310, the authenticity of the first information may be verified. In an embodiment, the processor 204 may be configured to verify the authenticity of the first information associated with the first document 116A, the first profile information 118A (i.e., the output of operation 302), or the set of filtered named entities associated with the first candidate (i.e., the set of named entities received at operation 306). The verification of the authenticity of the first information may be based on one or more public information databases (e.g., one or more public information DBs 108). The verification of the authenticity of the first information is further described, for example, in FIG. 5.

[0054] At 312, fairness associated with accepting the first candidate may be evaluated. In an embodiment, the processor 204 may be configured to evaluate fairness associated with accepting the first candidate. The processor 204 may be configured to evaluate a criterion that may correspond to a counterfactual criterion (e.g., a counterfactual fairness criterion) associated with the first candidate. The counterfactual criterion may correspond to a criterion such that a first acceptance probability of a first candidate having a first value of a first protected feature of the one or more protected features (i.e., in a first set of features) may be the same as a second acceptance probability of a second candidate having a second value of the first protected feature. The criterion may be based on a determination that other values ​​of the first set of features are the same. Examples of the one or more protected features may include at least one of age, gender, or a physical disability associated with the first candidate.

[0055] At 314, an empathy score associated with the first candidate may be evaluated. In an embodiment, processor 204 may be configured to evaluate an empathy score associated with the first candidate based on the output from operations 308, 310, and 312. Processor 204 may be configured to determine a third feature set based on differences (or deviations between) corresponding features from the first feature set associated with the first candidate and the second feature set associated with the candidate population (obtained at operation 308). Processor 204 may be configured to apply a pre-trained neural network model (e.g., neural network model 216) to the determined third feature set. Neural network model 216 may be pre-trained to determine empathy scores for different candidates based on the first feature sets associated with such candidates. Training of neural network model 216 (prior to determining the empathy scores) is further described, for example, with reference to FIG. 8. Further, the processor 204 may be configured to determine a set of weights associated with the third set of features based on application of the pre-trained neural network model 216. The processor 204 may be configured to determine an empathy score associated with the first candidate based on the determined set of weights.

[0056] With respect to verifying the authenticity of the first information (i.e., output of operation 310), in an embodiment, processor 204 may terminate the evaluation or determination of the empathy score of the first candidate if the first information associated with the first candidate is determined to be invalid (i.e., uncertain). In such a case, the first candidate's candidature for the job may be rejected. Such a validity check prior to evaluation of the empathy score associated with the first candidate may ensure that the details provided by the first candidate (e.g., but not limited to, education, certifications, and skill sets) are true, valid, and not misleading or falsified.

[0057] In an embodiment, processor 204 may be configured to apply a regularization constraint to a pre-trained neural network model (e.g., neural network model 216) based on a criterion associated with one or more protected features from the first feature set associated with the first candidate. The criterion may correspond to a counterfactual criterion (e.g., a counterfactual fairness criterion, i.e., the output of operation 312). The determination of an empathy score associated with the first candidate may be further based on the application of the regularization constraint to the pre-trained neural network model (e.g., neural network model 216). The evaluation of the counterfactual criterion and the application of the regularization constraint based on the counterfactual criterion are further described, for example, in FIG. 7.

[0058] The processor 204 may be further configured to render the determined empathy score associated with the first candidate. The automatic empathic assessment of job candidates is further described, for example, in Figures 4 and 9. An example scenario for determining empathy scores for two candidates is provided in Figure 10. An example scenario for a job description and a job candidate's resume is provided in Figure 11.

[0059] FIG. 4 depicts a flowchart of an example method for empathetic evaluation of job candidates, arranged in accordance with at least one embodiment described in the present disclosure. FIG. 4 is described in conjunction with elements from FIGS. 1, 2, and 3. With reference to FIG. 4, a flowchart 400 is shown. The method depicted in flowchart 400 may begin at 402 and may be performed by any suitable system, apparatus, or device, such as the example electronic device 102 of FIG. 1 or the processor 204 of FIG. 2. Although depicted as separate blocks, steps and operations associated with one or more of the blocks of flowchart 400 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0060] At block 402, first information regarding a first set of features associated with the first candidate may be extracted from at least one of documents or profile information associated with the first candidate. In an embodiment, the processor 204 may be configured to extract the first information regarding the first set of features associated with the first candidate from at least one of documents or profile information associated with the first candidate. As an example, the processor 204 may obtain first candidate information 120A associated with the first candidate. The first candidate information 120A may include a first document 116A, such as a resume associated with the first candidate. The first candidate information 120A may further include information related to first profile information 118A, such as, but not limited to, at least one of a job cover letter, a set of recommendations, a source code repository, a project page, a professional networking webpage, or a social networking webpage associated with the first candidate. The processor 204 may extract the first candidate information 120A from the candidate information DB 104 and store the extracted first candidate information 120A in the memory 206 and / or the persistent data storage 208. Alternatively, the processor 204 may obtain the first candidate information 120A based on a user input received from the user end device 110.

[0061] In an embodiment, the processor 204 may be configured to extract a first set of named entities from at least one of documents (e.g., the first documents 116A) or profile information (e.g., the first profile information 118A) associated with the first candidate. The processor 204 may be further configured to filter the extracted first set of named entities to extract a second set of named entities. Furthermore, the processor 204 may be configured to extract a first set of features associated with the first candidate based on excluding the extracted second set of named entities from the extracted first set of named entities. Examples of the first set of features may include, but are not limited to, at least one of a set of educational degrees, a set of certifications, years of work experience, location, a set of languages ​​studied or known, a physical disability, a set of skills, age, gender, or household / capita income associated with the first candidate. First information regarding the first set of features may thereby be extracted. The extracted first information (e.g., extracted from the first document 116A or from the first profile information 118A) may include various textual information or numerical values ​​related to each of the set of first features. For example, initial information for an education feature may indicate that the first candidate is likely to be a graduate student or have a master's degree. Similarly, initial information for a certification feature may indicate the number of certifications or the names of the certifications achieved by the first candidate. In another example, initial information for a language feature may indicate that the first candidate is likely to be a foreign language expert. Extraction of first information is further described, for example, in FIG. 5.

[0062] At block 404, second information regarding a second set of features corresponding to the first set of features may be extracted from one or more databases (e.g., one or more knowledge DBs 106). In an embodiment, processor 204 may be configured to extract second information regarding the second set of features corresponding to the first set of features from one or more databases (e.g., one or more knowledge DBs 106). In an embodiment, to extract the second information, processor 204 may be configured to send one or more queries to one or more databases (i.e., one or more knowledge DBs 106) based on one or more combinations of the first set of features associated with the first candidate. Based on the sent one or more queries, processor 204 may be configured to extract second information regarding the second set of features. The second set of features may be associated with a population of candidates likely to have at least one of the set of demographic parameters that is the same as the demographic parameters of the first candidate. Examples of the set of demographic parameters may include, but are not limited to, household income, location, type of institution studied, educational background, certifications, skill set, background, available resources, and languages ​​known or known. For example, the candidate population may include candidates in the same location or region as the first candidate. In another example, the candidate population may include candidates with similar household incomes (e.g., the candidate's total household income may be substantially close to or similar to that of the first candidate). In another example, the candidate population may include candidates who may be studying at the same type (or stage) of institution or who may have similar qualifications, which may be the same as that of the first candidate for whom an empathy score is to be determined. Examples of the set of second features may include, but are not limited to, at least one of the following: average per capita income, set of educational degrees, set of certifications, number of job applications, number of graduate students, number of institutions, number of secured jobs, or set of languages ​​known or studied, associated with the candidate population. Extraction of second information related to the set of second features is further described, for example, in FIG. 6.

[0063] At block 406, a third feature set may be determined based on differences between corresponding features from the first feature set associated with the first candidate and the second feature set associated with the candidate population. In an embodiment, the processor 204 may be configured to determine the third feature set based on differences between (i.e., deviations between) corresponding features from the first feature set associated with the first candidate and the second feature set associated with the candidate population. By way of example, the first feature set associated with the first candidate may be represented by a first feature vector, e.g., c=[c1, c2, c3, . . . c n ], and c i may represent individual features in the first feature set associated with the first candidate. Additionally, a second feature set associated with the population of candidates may be represented by a second feature vector, e.g., p=[p1, p2, p3, p n ], and p i may represent individual features in the second feature set associated with the candidate population. The processor 204 may be configured to normalize each feature in the first feature set and the second feature set based on the encoding to represent the particular feature in a corresponding vector format. For example, categorical features (e.g., educational background set, certification set, and skill set) may be converted into a one-hot vector.

[0064] Individual features in the second feature set may be represented by a second feature vector (i.e., p), such that each individual feature in the second feature set may be aligned with a corresponding feature in the first feature set represented by the first feature vector (i.e., c). As an example, in the first feature vector c, associated with the first candidate, c1 may represent a set of educational degrees, c2 may represent a set of certifications, c3 may represent a set of languages ​​studied / known, c4 may represent a set of skills, and so on. n may represent household income. Furthermore, in the second feature vector p, p1 may represent the average educational background, p2 may represent the average certification, p3 may represent the average set of languages ​​studied / known, and so on, relative to the candidate population. nmay represent the average household income. The processor 204 may be configured to determine a third set of features as the difference (or deviation) between corresponding features in the first and second feature vectors. As an example, given a third feature vector m=[m1, m2, m3, m n ] may be determined as m=cp (i.e., m i =c i -p i ). In an embodiment, the greater the difference between corresponding / related features in the first feature vector (i.e., for the first candidate) and the second feature vector (i.e., for the population of candidates), the greater the skill acquisition ability or diligence of the first candidate relative to the population of candidates. For example, if the first candidate has an undergraduate degree (i.e., one first piece of information in the first set of features) and the average education of the population of candidates (e.g., in the same region of the first candidate) is higher education (i.e., one second piece of information in the second set of features), the greater the difference or deviation between the corresponding features will be, indicating that the first candidate has a superior skill set and may be a very diligent worker or may have studied exceptionally hard to achieve the undergraduate skills, compared to the population of candidates with the same demographics (e.g., the same region). Therefore, in such a case, the empathy score or weighting assigned to the first candidate will be higher. Examples of empathy scores for different candidates may be provided, for example, in FIGS. 10 and 11.

[0065] At block 408, a pre-trained neural network model (e.g., neural network model 216) may be applied to the determined third feature set. In an embodiment, processor 204 may be configured to apply the pre-trained neural network model to the determined third feature set (represented by a third feature vector m). As an example, processor 204 may input the third feature vector (i.e., m) to an input layer of neural network model 216 to apply neural network model 216 to the determined third feature set. In an embodiment, neural network model 216 may be pre-trained to determine an empathy score for each candidate set based on a first feature set associated with the candidate set. The first feature set associated with the candidate set may be a training dataset for neural network model 216. Training of neural network model 216 is further described, for example, with reference to FIG. 8.

[0066] At block 410, a set of weights associated with the third feature set may be determined based on application of the pre-trained neural network model 216. In an embodiment, the processor 204 may be configured to determine the set of weights associated with the third feature set based on application of the pre-trained neural network model 216. As an example, the pre-trained neural network model 216 may be configured to weight the individual features (i.e., m) represented by the third feature vector (i.e., m) to obtain an empathy score associated with the first candidate. i ) The pre-trained neural network model 216 may be configured to learn a convex combination of each individual feature of the third feature set (i.e., m i ) and determine the value of the corresponding weight accordingly. For example, a higher weight may be assigned to a more important feature, etc. The individual features (i.e., m) in the third feature set (i.e., m) that may be learned by the pre-trained neural network model 216 may be weighted accordingly. i) can be expressed by equation (1) as follows:

number

[0067] In an embodiment, the set of weights associated with the third feature set may be determined based on the output of a penultimate layer of the pre-trained neural network model 216. The penultimate layer of the pre-trained neural network model 216 may output a vector of real-valued scores. Each weight in the set of weights associated with the third feature set may be determined based on the real-valued scores that may be output from each node of the penultimate layer of the pre-trained neural network model 216.

[0068] At block 412, an empathy score associated with the first candidate may be determined based on the determined set of weights. In an embodiment, the processor may be configured to determine an empathy score associated with the first candidate based on the determined set of weights. For example, the final layer of the pre-trained neural network model 216 may include a normalization layer, such as a softmax layer. The pre-trained neural network model 216 may determine the empathy score associated with the first candidate in the final layer based on normalizing the weighted sum of the third feature vector (based on the weighted sum in Equation (1)).

[0069] At block 414, the determined empathy score associated with the first candidate may be rendered. In an embodiment, the processor 204 may be configured to render the determined empathy score associated with the first candidate. The processor 204 may display the determined empathy score associated with the first candidate along with the first candidate information 120A (including the first document 116A and the first profile information 118A) to the user 114 on the display screen 212. The displayed empathy score, along with the first candidate information 120A associated with the first candidate, may enable evaluation of the first candidate for the job. One or more other automated techniques / tools may also be used, or manual analysis may be performed to evaluate the first candidate based on the determined empathy score and the first candidate information 120A. Control may then pass to an end.

[0070] Although flowchart 400 is depicted as separate operations such as 402, 404, 406, 408, 410, 412, and 414, in particular embodiments, such separate operations may be separated into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0071] FIG. 5 depicts a flowchart of an example method for extracting and verifying first information associated with a first candidate, arranged in accordance with at least one embodiment described herein. FIG. 5 is described in conjunction with elements from FIGS. 1, 2, 3, and 4. With reference to FIG. 5, a flowchart 500 is shown. The method depicted in flowchart 500 may begin at 502 and may be performed by any suitable system, apparatus, or device, such as the example electronic device 102 of FIG. 1 or the processor 204 of FIG. 2. Although depicted as separate blocks, steps and operations associated with one or more of the blocks of flowchart 500 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0072] At block 502, a first set of named entities may be extracted from at least one of a document (e.g., the first document 116A) or profile information (e.g., the first profile information 118A) associated with the first candidate. In an embodiment, the processor 204 may be configured to extract the first set of named entities from at least one of a document (e.g., the first document 116A) or profile information (e.g., the first profile information 118A) associated with the first candidate. Examples of techniques or tools that may be used for the extraction of the first set of named entities from the first document 116A or the first profile information 118A may include, but are not limited to, Nlp.js, spark.js, Stanford NLPcore, or Tensorflow sequence tagging. Examples of the extracted set of named entities may include, but are not limited to, education, certifications, location, marital status, languages ​​known / studied, disability, skill set, age, gender, and household / capita income associated with the first candidate.

[0073] At block 504, the extracted first set of named entities may be filtered to extract a second set of named entities. In an embodiment, a processor may be configured to filter the extracted first set of named entities to extract a second set of named entities. In an embodiment, the filtering of the extracted first set of named entities may be based on predetermined rules or user input (i.e., received from the user 114). Examples of techniques or tools that may be used for filtering the extracted first set of named entities may include, but are not limited to, Stanford NLPcore or Tensorflow sequence tagging. In an example, the marital status of a first candidate may be filtered from the example first set of named entities (described at block 502) to extract the second set of named entities.

[0074] At block 504, a first set of features associated with the first candidate may be extracted based on the exclusion of the extracted second set of entities from the extracted first set of entities. In an embodiment, the processor 204 may be configured to extract a first set of features associated with the first candidate based on the exclusion of the extracted second set of entities from the extracted first set of entities. In an example, as described at block 504, for example, the marital status of the first candidate (i.e., extracted as the second set of entities) may be filtered from the first set of entities. That is, the second set of entities (i.e., the entity "marriage status") may be excluded from the first set of entities. The exclusion of the second set of entities may ensure that information that is not relevant to (or does not affect) the empathy score (e.g., marital status) can be excluded from the empathy score determination process. This may avoid bias and unexpected results (e.g., biasing based on marital status). Based on the exclusion of the second set of entities from the first set of entities, a first set of features may be extracted, thereby extracting first information about the first set of features. Examples of the first set of features may include at least one of a set of educational degrees, a set of certifications, years of work experience, location, a set of languages ​​studied or known, a physical disability, a set of skills, age, gender, or per capita income associated with the first candidate.

[0075] At block 508, the reliability of first information associated with at least one of a document, profile information, or first set of named entities associated with the first candidate may be verified. In an embodiment, the processor 204 may be configured to verify the reliability of first information associated with at least one of a document (e.g., first document 116A), profile information (e.g., first profile information 118A), or first set of named entities associated with the first candidate. Verification of the reliability of the first information may be based on relevant information extracted or checked from one or more public information databases (e.g., one or more public information DBs 108). For example, the processor 204 may verify the reliability of information regarding educational degrees and certifications associated with the first candidate (i.e., the first information) from one or more public information DBs 108. Examples of such public information DBs 108 may include, but are not limited to, college / university records or databases or school education record databases. The reliability of educational details associated with the first candidate, such as completing a course, obtaining a specific grade, or obtaining a specific certification, may be verified based on the college / university records or database or the school's educational record database. Furthermore, the processor 204 may verify the reliability of information related to the first candidate's work experience and skill set from one or more public information DBs 108. Examples of such one or more public information DBs 108 may include, but are not limited to, human resources (HR) databases or job portal databases of organizations where the first candidate has worked. In an example, the one or more public information DBs 108 for verifying language, age, or gender may include police station records, birth-related department records, or department records that can track people's locations for security and surveillance purposes. In an embodiment, the processor 204 may be configured to transmit the first information related to each of the set of first characteristics to a computing device or server associated with one or more public information DBs 108 and further receive information related to the verification of the transmitted first information from the corresponding one or more public information DBs 108.

[0076] In embodiments, the determination of the empathy score associated with the first candidate may be further based on verifying the authenticity of the first information. In embodiments, the processor 204 may terminate the evaluation or determination of the empathy score of the first candidate if the first information associated with the first candidate is determined to be invalid or uncertain (e.g., as described in FIGS. 3 and 4). In such cases, the first candidate's candidacy for the job may be rejected. Such a validity check prior to the evaluation of the empathy score associated with the first candidate may ensure that the details (e.g., education, work experience, certifications, skill set, age, gender, or location) supplied by the first candidate are true and not misleading or falsified. Control may be over the termination.

[0077] Although flowchart 500 is depicted as separate operations such as 502, 504, 506, and 508, in particular embodiments, such separate operations may be separated into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0078] Figure 6 depicts a flowchart of an example method for extracting second information related to a candidate population, arranged in accordance with at least one embodiment described in the present disclosure. Figure 6 is described in conjunction with elements from Figures 1, 2, 3, 4, and 5. With reference to Figure 6, a flowchart 600 is shown. The method depicted in flowchart 600 may begin at 602 and may be performed by any suitable system, apparatus, or device, such as the example electronic device 102 of Figure 1 or the processor 204 of Figure 2. Although depicted as separate blocks, steps and operations associated with one or more of the blocks of flowchart 600 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0079] At block 602, one or more queries may be sent to one or more databases (e.g., one or more knowledge DBs 106). In an embodiment, the processor 204 may be configured to send one or more queries to one or more databases (e.g., one or more knowledge DBs 106). The one or more knowledge DBs 106 may include, but are not limited to, a census database, a local government database, an economic survey database, or a job portal database. The one or more queries may be formed based on one or more combinations of a first set of features associated with a first candidate. For example, the one or more queries may be formed based on one or more combinations of pairs of features from the first set of features associated with the first candidate. As an example, a first query from the one or more queries may be formed based on a first combination of the first set of features, such as the location and institution of the first candidate. In another example, a second query from the one or more queries may be formed based on a second combination of the first set of features, such as the gender and location, including subdivision (or local government area), of the first candidate. In a scenario where the first feature set includes five features and one or more queries are formed based on one or more combinations of pairs of features in the first feature set, then ten different queries (i.e., 5 C2=10) may be formed as one or more queries. Forming multiple queries based on one or more combinations of features in the first feature set may increase the reliability and robustness of the extraction of the second information. In an embodiment, the transmitted queries may correspond to search queries, and based on these queries, one or more knowledge DBs 106 may search or retrieve information (e.g., second information related to the second feature set) based on the received queries.

[0080] At block 604, second information regarding a second set of features corresponding to the first set of features may be extracted based on the one or more queries sent. In an embodiment, the processor 204 may be configured to extract the second information regarding the second set of features based on one or more queries sent to one or more knowledge DBs 106. The second set of features may be associated with a population of candidates having at least one of the same set of demographic parameters as the demographic parameters associated with the first candidate. Examples of the set of demographic parameters may include, but are not limited to, household income, location, type of institution studied, educational background, certifications, skill set, family background, available resources, and languages ​​known or studied. For example, the population of candidates may be from the same location as the first candidate and may have studied at the same educational institution or the same type of educational institution (e.g., at the same stage). In such an example, the location and education (or educational institution) from the first feature set may be formed as a query that can be sent to one or more knowledge DBs 106 to further extract second information (i.e., information about a population of candidates that may match the submitted query). The second information for each of the second feature sets may correspond to or be similar to the first feature set of the first candidate. That is, all features extracted from the candidate information DB 104 for the first candidate may be extracted from one or more knowledge DBs 106 for the candidate population, where one or more features of the candidate population may match or be retrieved based on the submitted query. In another example, the candidate population may be from the same location as the first candidate and belong to the same subregion (or local government area). The second feature set may include, but is not limited to, at least one of the following related to the candidate population: average per capita income, set of educational degrees, set of certifications, number of job applications, number of graduate students, number of institutions, number of secured jobs, and set of languages ​​studied or known. Thus, one or more knowledge DBs 106 may provide population-level statistics for particular demographics (first choice locations, institutions, ages, etc.).Here, population-level statistics may include information related to, but not limited to, the average employment rate, average household income, number of available educational institutions, average level of education, and average number of languages ​​studied / known by a population of candidates with the same demographics as the first candidate. That is, the second information extracted from one or more knowledge DBs 106 may provide average statistics related to a population (or collection) of candidates who may have the same demographics (same location, e.g., city or state) as the first candidate for whom an empathy score is to be determined. Furthermore, different queries (i.e., queries formed based on different combinations of features in the first feature set) may target different segments of the candidate population, and further relevant second information related to the targeted population may be retrieved from one or more knowledge DBs 106. This may further enhance the robustness of the disclosed system to target relevant populations of candidates to be compared to the first candidate for determining an empathy score. A difference or comparison between the first feature set of the first candidate and the second feature set for the population of candidates may cause control to pass to an end, for example, at block 406 of FIG. 4, as further described.

[0081] Although flowchart 600 is depicted as separate operations such as 602, 604, etc., in particular embodiments, such separate operations may be separated into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0082] FIG. 7 depicts a flowchart of an example method for empathy score determination based on counterfactual criteria, arranged in accordance with at least one embodiment described in the present disclosure. FIG. 7 is described in conjunction with elements from FIGS. 1, 2, 3, 4, 5, and 6. With reference to FIG. 6, a flowchart 600 is shown. The method depicted in flowchart 600 may begin at 602 and may be performed by any suitable system, apparatus, or device, such as the example electronic device 102 of FIG. 1 or the processor 204 of FIG. 2. Although depicted as separate blocks, steps and operations associated with one or more of the blocks of flowchart 600 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0083] At block 702, a regularization constraint may be applied to the pre-trained neural network model 216 based on a criterion associated with one or more protected features from the first feature set associated with the first candidate. In an embodiment, the processor 204 may be configured to apply the regularization constraint to the pre-trained neural network model 216 based on a criterion associated with one or more protected features from the first feature set associated with the first candidate. In an embodiment, the criterion may correspond to a counterfactual criterion (e.g., a counterfactual fairness criterion) that a first acceptance probability of a first candidate having a first value of a first protected feature of the one or more protected features is the same as a second acceptance probability of a second candidate having a second value of the first protected feature. The criterion may be based on a determination that other values ​​of the first feature set of the first candidate and the second candidate are the same. Examples of the one or more protected features may include at least one of age, gender, or physical disability associated with the first candidate. The counterfactual criterion can be expressed in equation (2) as follows, for every value a' reachable by A and for every value of y:

number

[0084] In an example, consider gender as a first protected characteristic of a first candidate. Based on the counterfactual criterion, given that other characteristics in the first characteristic sets of the first and second candidates are the same, the probability (i.e., first likelihood) of selecting the first candidate (i.e., if the first candidate is a female candidate (i.e., first value)) is equal to the probability (i.e., second likelihood) of selecting the second candidate (i.e., if the second candidate is a male candidate (second value)). The counterfactual criterion may be applied as a regularization constraint to the pre-trained neural network model 216. Applying the counterfactual criterion as a regularization constraint may ensure that the determination of the empathy score of the first candidate is unbiased with respect to one or more protected characteristics of the first candidate.

[0085] At block 704, an empathy score associated with the first candidate may be determined further based on application of a regularization constraint to the pre-trained neural network model 216. In an embodiment, the processor 204 may be configured to determine an empathy score associated with the first candidate further based on application of a regularization constraint to the pre-trained neural network model 216. The processor 204 may use Equation (1) (i.e., learned by the pre-trained neural network model 216, as described in FIG. 4 ) and Equation (2) to determine the empathy score based on Equation (3), as follows:

number

[0086] Thus, based on equation (3), the counterfactual criterion may be incorporated into the empathy score determination as a regularization constraint applied to the pre-trained neural network model 216, and the empathy score associated with the first candidate may be determined accordingly. Control may pass to termination.

[0087] Although flowchart 700 is depicted as separate operations such as 702, 704, etc., in particular embodiments, such separate operations may be separated into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0088] FIG. 8 depicts a flowchart of an example method for training a neural network model based on each of a set of documents assigned an empathy score by a set of raters, arranged in accordance with at least one embodiment described herein. FIG. 8 is described in conjunction with elements from FIGS. 1, 2, 3, 4, 5, 6, and 7. With reference to FIG. 8, a flowchart 800 is shown. The method depicted in flowchart 800 may begin at 802 and may be performed by any suitable system, apparatus, or device, such as the example electronic device 102 of FIG. 1 or the processor 204 of FIG. 2. Although depicted as separate blocks, steps and operations associated with one or more of the blocks of flowchart 800 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0089] At block 802, a set of empathy scores may be received for a set of documents in the training dataset of the neural network model 216 based on user input from one or more raters. In an embodiment, the processor 204 may be configured to receive a set of empathy scores for a set of documents in the training dataset of the neural network model 216 based on user input from one or more raters. The set of documents may include a set of resumes or résumés associated with a set of candidates (i.e., different from the first candidate for whom an empathy score is to be determined). The user input from the one or more raters may include tags or labels assigned to each of the set of documents. The tags or labels assigned to the candidate documents may correspond to empathy scores assigned to the corresponding candidate by a particular rater based on information present in the candidate documents. However, the empathy scores (i.e., tags) assigned to each of the set of documents (i.e., manually by one or more raters) may be prone to bias, in that the rater's personal opinions may be subject to bias or uncertainty. In an embodiment, the processor 204 may be configured to aggregate the personal opinions (i.e., empathy scores assigned to a set of documents) associated with each of the set of raters based on an optimal fusion technique (e.g., as described in the following operational steps 804-806 in FIG. 8).

[0090] At block 804, pairwise rankings may be calculated among the set of received empathy scores for each in the set of documents. In an embodiment, the processor 204 may be configured to calculate pairwise rankings among the set of received empathy scores for each in the set of documents. For example, let "A" represent the set of "m" choices of empathy scores assigned to the documents by the set of raters. Further, (outside 1) Let TIFF0007753929000004.tif17170 represent a set of rankings of options. (outside 2) For TIFF0007753929000005.tif16170, (Outside 3) TIFF0007753929000006.tif18170 represents that a first choice "a" (i.e., a first value of the empathy score for the document) is likely to be preferred (or have a higher ranking) to a second choice "b" (i.e., a second value of the empathy score for the document) within "σ." In an embodiment, a pairwise ranking between the first choice "a" and the second choice "b" may be calculated based on the number of raters who assign the first value of the empathy score corresponding to the first choice "a" and the number of raters who assign the second value of the empathy score corresponding to the second choice "b."

[0091] At block 806, an optimal empathy score ranking may be determined for each of the set of documents based on minimizing the Kendall tau distance across the calculated pairwise rankings. In an embodiment, the processor 204 may be configured to determine an optimal empathy score ranking for each of the set of documents based on minimizing the Kendall tau distance across the calculated pairwise rankings. For example, a rater's (i.e., rater "i") uncertain input preferences (i.e., empathy scores assigned to documents) may be multiplied by the rater's actual preferences (i.e., σ i ) may be distributed across the rankings that may be derived. (outside 4) Let TIFF0007753929000007.tif18170 represent the set of distributions over the rankings of the options in "A". (outside 5) A voting rule such as TIFF0007753929000008.tif17170 (where n may be the number of raters in the rater set) may be used to aggregate the uncertain empathy scores that may be assigned to documents by the rater set into a social ranking, where the objective function (i.e., h(σ)) may be expressed by Equation (4) as follows:

number

[0092] The objective function expressed in equation (4) may be simplified and expressed by equations (5A) and (5B) as follows:

number

[0093] The objective function expressed in equation (4) and simplified in equations (5A) and (5B) may be the expected sum of Kendall distances from the uncertain input preferences of the set of evaluators. In equation (5A), (outside 8) TIFF0007753929000013.tif19170 is (outer 9) TIFF0007753929000014.tif21170, which may be the probability that the evaluator prefers the second option "b" over the first option "a". The processor 204 determines the optimal ranking based on minimizing an objective function (i.e., σ OPT ) may be determined. The optimal ranking may be expressed by equation (6) as follows:

number

[0094] The optimal ranking represented by equation (6) may be the minimal ranking associated with the uncertain input preferences (e.g., empathy scores assigned to documents) of the set of raters. The optimal ranking may thereby be the optimal empathy score ranking.

[0095] At block 808, the neural network model 216 may be trained based on the determined optimal empathy score ranking for each of the document sets in the training dataset. In an embodiment, the processor 204 may be configured to train the neural network model 216 based on the determined optimal empathy score ranking for each of the document sets in the training dataset. For example, the processor 204 may determine an optimal value of the empathy score for each of the document sets in the training dataset based on the determined optimal empathy score ranking. Further, the processor 204 may be configured to extract a first set of features associated with each of the candidate sets based on the document sets. Extracting the first set of features based on the documents is described, for example, in FIG. 5. Based on the optimal value of the empathy score for each of the document sets in the training dataset and the determined first feature set for each of the candidate sets, the processor 204 may train the neural network model 216 to determine an empathy score associated with the candidate using the candidate's first feature set. Control may pass to end. Training the neural network model 216 may be described, for example, in FIG. 1.

[0096] Although flowchart 800 is depicted as separate operations such as 802, 804, 806, and 808, in particular embodiments, such separate operations may be separated into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0097] Figure 9 is a diagram depicting an example scenario of determining an empathy score associated with a first candidate, arranged in accordance with at least one embodiment described in the present disclosure. Figure 9 will be described in conjunction with elements from Figures 1, 2, 3, 4, 5, 6, 7, and 8. With reference to Figure 9, an example scenario 900 is shown. The example scenario 900 includes application-specific results 902, knowledge database query results 904, a neural network model 906, labeled data 908, and fairness constraints 910.

[0098] The application-specific results 902 may include first information regarding a first set of features associated with the first candidate. The processor 204 may extract the application-specific results 902 from the first candidate information 120A, which includes the first document 116A and the first profile information 118A associated with the first candidate. Extraction of the application-specific results 902 is described, for example, in Figures 3, 4, and 5.

[0099] The knowledge database query results 904 may include second information about a second set of features corresponding to the first set of features. The second set of features may be associated with a population of candidates having at least one of the set of demographic parameters that is the same as the demographic parameters of the first candidate. The processor 204 may extract the knowledge database query results 904 (i.e., second information about the second set of features) from one or more knowledge DBs 106 based on one or more queries formed from one or more combinations of the first set of features. Extraction of the knowledge database query results 904 based on one or more submitted queries is described, for example, in Figures 3, 4, and 5.

[0100] The neural network model 906 may correspond to the neural network model 216. The neural network model 906 may be trained to determine empathy scores associated with the candidates based on labeled data 908. The labeled data 908 may be received from a set of raters. The labeled data 908 may include sets of documents associated with the set of candidates based on user input from the set of raters, assigned empathy scores (i.e., assigned by the set of raters) for the sets of documents in the set of candidates, and optimal empathy scores assigned to each of the set of candidates. The training of the neural network model 906 may be based on an optimal fusion technique, for example, as described in FIG. 8.

[0101] The fairness constraint 910 may include a criterion associated with one or more protected features from a first feature set associated with a first candidate. The criterion may correspond to a counterfactual criterion such that a first acceptance probability of a first candidate having a first value of a first protected feature among the one or more protected features is the same as a second acceptance probability of a second candidate having a second value of the first protected feature. The criterion may be based on a determination that other values ​​in the first feature set are the same.

[0102] In an embodiment, the processor 204 may be configured to determine a difference between the application-specific results 902 and the knowledge database query results 904, as shown in FIG. 9. The difference may correspond to a third feature set. Determining the third feature set is described, for example, in FIGS. 3 and 4. The processor 204 may feed the difference to a pre-trained neural network model 906 to apply the neural network model 906 to the third feature set. As described above, the neural network model 906 may be trained based on the labeled data 908. The processor 204 may determine a set of weights associated with the third feature set based on the application of the pre-trained neural network model 906. The processor 204 may be configured to determine an empathy score associated with the first candidate based on the determined set of weights. In an embodiment, the processor 204 may be configured to apply a fairness constraint 910 as a regularization constraint to the pre-trained neural network model 906. The determination of the empathy score associated with the first candidate may be further based on application of a regularization constraint (i.e., a fairness constraint 910) to the pre-trained neural network model 906. The processor 204 may be configured to render the determined empathy score associated with the first candidate. The determination of the empathy score is further described, for example, in Figures 3, 4, and 7.

[0103] It may be noted that the scenario 900 shown in FIG. 9 is presented merely as an example and should not be construed as limiting the scope of the present disclosure.

[0104] FIG. 10 is a diagram depicting an example scenario of empathy scores that may be determined for two candidates, arranged in accordance with at least one embodiment described in the present disclosure. FIG. 10 will be described in conjunction with elements from FIGS. 1, 2, 3, 4, 5, 6, 7, 8, and 9. With reference to FIG. 10, an example scenario 1000 is shown. The example scenario 1000 includes applicant 1 1002A, applicant 2 1002B, first information 1004A associated with applicant 1 1002A, first information 1004B associated with applicant 2 1002B, an empathy score 1006A associated with applicant 1 1002A, and an empathy score 1006B associated with applicant 2 1002B.

[0105] 10 , first information 1004A associated with applicant 1 1002A may include details such as, but not limited to, background (e.g., low-income family), type of educational institution (e.g., government school in a developing country), and region / location (e.g., developing country) associated with applicant 1 1002A. Similarly, as shown in FIG. 10 , first information 1004B associated with applicant 2 1002B may include details such as, but not limited to, background (e.g., middle-class family), type of educational institution (e.g., private school in a developing country), and region / location (e.g., developing country) associated with applicant 2 1002B. Empathy score 1006A associated with applicant 1 1002A, which may be determined based on first information 1004A, may have a higher value than empathy score 1006B associated with applicant 2 1002B, which may be determined based on first information 1004B. For example, the empathy score 1006A associated with Applicant 1 1002A may have a value of 0.9, while the empathy score 1006B associated with Applicant 2 1002B may have a value of 0.5. This may be because, although both applicants may have the same level of education, Applicant 1 1002A may be from a lower-income family and may have graduated from a government school, compared to Applicant 2 1002B, who may be from a middle-class family and may have graduated from a private school. Thus, Applicant 1 1002A may have a higher skill acquisition ability and work harder than Applicant 2 1002B, thereby causing the empathy score 1006A associated with Applicant 1 1002A to be higher than the empathy score 1006B associated with Applicant 2 1002B.

[0106] It may be noted that the scenario 1000 shown in FIG. 10 is presented merely as an example and should not be construed as limiting the scope of the present disclosure.

[0107] Figure 11 is a diagram depicting an example scenario of a job description for a job and a resume for a job candidate, arranged in accordance with at least one embodiment described in the present disclosure. Figure 11 will be described in conjunction with elements from Figures 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. With reference to Figure 11, an example scenario 1100 is shown. The example scenario 1100 includes a job description 1102 associated with a job and a resume 1104 associated with a job candidate.

[0108] The job description 1102 may include details related to job-specific requirements. As shown in FIG. 11 , such details in the job description 1102 may include, but are not limited to, job responsibilities / designation (e.g., marketing manager), desired education (e.g., master's degree in communications), desired languages ​​(e.g., English, French, and German), and desired skill set (e.g., ability to interact fluently with customers). Additionally, the resume 1104 may include details related to the candidate's educational background, skill set, and personal information. FIG. 11 illustrates that the candidate's details in the resume 1104 may include, but are not limited to, the candidate's associated name (e.g., "XYZ"), country (e.g., Nigeria), GPA score (e.g., 3.9), languages ​​known (e.g., English (basic level) and native language), awards (award in debate), educational institution (government school), and household income (e.g., less than US$50,000 per year).

[0109] Typically, traditional solutions for candidate evaluation may reject a job candidate based on the candidate's lack of the required educational background (i.e., a master's degree in communications) and language skills (e.g., French and German). However, the disclosed system for empathetic candidate evaluation may assign a high empathy score to a candidate based on details in the resume 1104 and demographic parameters or similar details related to the candidate's population within the same region (e.g., Nigeria) as the candidate. A high empathy score may indicate that the candidate may have a high skill acquisition ability and a hard work ethic. For example, a candidate's high GPA score may indicate that the candidate may be a quick learner who may have a good learning curve in grasping new skills related to the job. Furthermore, if the candidate has won awards in debates, the candidate may be fluent in communication and have excellent negotiation skills. Thus, a candidate may be competent, but may not be able to pursue a higher degree of communication due to a lack of resources in his / her location (e.g., Nigeria) and / or a low household income (e.g., less than US$50,000 per year). Similarly, a candidate may not be able to learn fluent English (or other desired language) due to his / her location (e.g., Nigeria). Thus, in such cases, the disclosed system may assign a high empathy score to the candidate, taking into account the candidate's high skill acquisition ability and diligence, compared to details of other candidates (i.e., the population) in the same region obtained from one or more knowledge DBs 106 (i.e., as described in block 404 of FIG. 4 and in FIG. 6 ). A high empathy score assigned to a candidate (i.e., by the disclosed system) may lead to the selection of a candidate who might otherwise be rejected for a job. Thus, the disclosed system may lead to an empathetic evaluation of candidates for a job.

[0110] It may be noted that the scenario 1100 shown in FIG. 11 is presented merely as an example and should not be construed as limiting the scope of the present disclosure.

[0111] The disclosed electronic device 102 may determine a difference between a first set of characteristics of a candidate (e.g., but not limited to, educational background, certifications, languages ​​known, and skill set) and corresponding characteristics in a second set of characteristics of a population of candidates belonging to the same demographic background as the candidate. An empathy score associated with the candidate may then be determined based on the difference between the first set of characteristics associated with the candidate and the second set of characteristics associated with the population of candidates. The difference (i.e., the third set of characteristics) may indicate a greater skill acquisition ability and diligence of the candidate compared to other candidates with similar demographic backgrounds. The greater the difference, the higher the empathy score associated with the candidate, and the higher the likelihood of the candidate being selected for the job. The disclosed electronic device 102 may thereby include empathy as one of the factors for automated evaluation of job candidates based on the difference between the candidate's qualifications or skill set and that of a population of candidates of similar demographic backgrounds to the candidate, compared to other conventional systems that may not consider such empathy in evaluating job candidates. The disclosed electronic device 102 may be advantageous in that a candidate's inherent skill acquisition ability and diligence, which may be useful for most positions, may be taken into account during the automated evaluation of the candidate based on the empathy score determined for the candidate. Thus, the individual constraints, needs, and differences of various candidates may be taken into account in determining the candidate's empathy score and evaluating the candidate's resume. The disclosed empathy-based candidate evaluation techniques may be applicable across different languages ​​(i.e., resumes in different languages), across various employment scenarios, and may be combined with existing automated, semi-automated, or manual resume evaluation techniques.

[0112] The disclosed electronic device 102 may determine a set of weights associated with the third feature set based on application of the third feature set to the neural network model 216 (i.e., as described, for example, in block 408 of FIG. 4 ). The electronic device 102 may then provide a reasoning associated with a particular empathy score for the candidate based on the set of weights associated with the third feature set, with which the neural network model 216 may be trained. The reasoning may indicate the contribution of each feature associated with the candidate in determining the particular empathy score. The set of weights may be used as a decision-making factor in selecting the candidate. Based on the candidate's empathy score, the candidate may also be given equal employment opportunities over other candidates with similar qualifications / skill sets but different demographic backgrounds. The disclosed empathic assessment of candidates may thus also help increase workforce diversity within an organization. Furthermore, by incorporating counterfactual fairness criteria as regularization constraints, the disclosed electronic device 102 may be less biased with respect to various protected or sensitive characteristics (e.g., age, disability, and gender) compared to conventional systems.

[0113] Various embodiments of the present disclosure may provide one or more non-transitory computer-readable storage media configured to store instructions that, when executed, cause a system (e.g., an exemplary electronic device 102) to perform operations. The operations may include extracting first information regarding a first set of features associated with a first candidate from at least one of documents or profile information associated with the first candidate. The operations may further include extracting second information regarding a second set of features corresponding to the first set of features from one or more databases. The second set of features may be associated with a population of candidates having at least one of the same set of demographic parameters as the first candidate. The operations may further include determining a third set of features based on differences in corresponding features from the first set of features associated with the first candidate and the second set of features associated with the population of candidates. The operations may further include applying a pre-trained neural network model to the determined third set of features. The neural network model may be pre-trained to determine an empathy score for each candidate in the set of candidates based on a first set of features associated with the set of candidates. The operations may further include determining a set of weights associated with a third set of features based on application of the pre-trained neural network model. The operations may further include determining an empathy score associated with the first candidate based on the determined set of weights. The operations may further include rendering the determined empathy score associated with the first candidate.

[0114] As used in this disclosure, the term "module" or "component" may refer to a specific hardware implementation configured to perform the operations of the module or component and / or a software object or software routine that may be stored on and / or executed by general-purpose hardware (e.g., computer-readable media, processing device, etc.) of a computing system. In some embodiments, different components, modules, engines, and services described in this disclosure may be implemented as objects or processes that execute on a computing system (e.g., as separate threads). While some of the systems and methods described in this disclosure are generally described as implemented in software (stored on and / or executed by general-purpose hardware), specific hardware implementations or combinations of software and specific hardware implementations are also possible and contemplated. As used herein, a "computing entity" may be any computing system as defined above in this disclosure, or any module or combination of modules executing on a computing system.

[0115] Terms used in this disclosure, particularly in the appended claims (e.g., the body of the appended claims), are generally intended to be “open” terms (e.g., the word “including” should be interpreted as meaning “including, but not limited to,” the word “having” should be interpreted as “comprising at least,” the word “includes” should be interpreted as “including, but not limited to,” etc.).

[0116] Furthermore, when a specific number is intended in an introduced claim recitation, such intention is clearly recited in the claim; otherwise, no such intention exists. For example, to facilitate understanding, the following appended claims may use introductory phrases such as "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be interpreted as suggesting that a particular claim containing such introduced claim recitation is limited to instances containing only one of the recited item, even if the same claim contains both an introductory phrase such as "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be interpreted to mean "at least one" or "one or more"). The same applies when introducing claim recitations using definite articles.

[0117] Furthermore, even when a specific number is explicitly stated in an introduced claim, those skilled in the art will understand that such a statement should generally be interpreted to mean at least the recited number (e.g., a statement simply stating "two items," without any other modifiers, means at least two items, or more than two items). Furthermore, when phrases like "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc." are used, such structure is generally intended to include A only, B only, C only, both A and B, both A and C, both B and C, and / or all of A, B, and C, etc.

[0118] Furthermore, any disjunctive word and / or disjunctive phrase presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to contemplate the possibility of including one of those terms, either of those terms, or both of those terms. For example, the phrase "A or B" should be understood to include the possibilities of "A or B" or "A and B."

[0119] All examples and conditional language set forth in this disclosure are intended for educational purposes to aid the reader in understanding the concepts and inventions contributed by the inventors to the advancement of the art, and should not be construed as being limited to such specifically set forth examples and conditions. Although embodiments of the present disclosure have been described in detail, various changes, substitutions, and alterations may be made thereto without departing from the spirit and scope of the present disclosure.

[0120] In addition to the above embodiments, the following supplementary notes are disclosed. (Appendix 1) 1. A processor-implemented method comprising: extracting first information regarding a first set of features associated with a first candidate from at least one of documents or profile information associated with the first candidate; extracting second information from one or more databases regarding a second set of features corresponding to the first set of features, the second set of features being associated with a population of candidates that includes at least one of the same set of demographic parameters as the demographic parameters of the first candidates; determining a third set of features based on corresponding feature differences from the first set of features associated with the first candidate and the second set of features associated with the population of candidates; applying a pre-trained neural network model to the determined third feature set, the neural network model being pre-trained to determine an empathy score for each of the candidate set based on the first feature set associated with the candidate set; determining a set of weights associated with the third feature set based on the application of the pre-trained neural network model; and determining an empathy score associated with the first candidate based on the determined set of weights; rendering the determined empathy score associated with the first candidate; and A method having the following. (Appendix 2) applying a regularization constraint to the pre-trained neural network model based on a criterion associated with one or more protected features from the first feature set associated with the first candidate; determining an empathy score associated with the first candidate is further based on applying the regularization constraint to the pre-trained neural network model. The method described in Appendix 1. (Appendix 3) the criterion corresponds to a counterfactual criterion that a first acceptance probability of the first candidate having a first value of a first protected feature of the one or more protected features is the same as a second acceptance probability of a second candidate having a second value of the first protected feature, based on a determination that other values ​​of the first set of features are the same. The method described in Appendix 2. (Appendix 4) the one or more protected characteristics include at least one of age, sex, or a disability associated with the first candidate; The method described in Appendix 2. (Appendix 5) the profile information associated with the first candidate includes information associated with at least one of a job cover letter, a set of recommendations, a source code repository, a project page, a professional networking web page, or a social networking web page associated with the first candidate; The method described in Appendix 1. (Appendix 6) extracting a first set of named entities from at least one of the documents or the profile information associated with the first candidate. The method described in Appendix 1. (Appendix 7) filtering the extracted first set of named entities to extract a second set of named entities; extracting the first set of features associated with the first candidate based on exclusion of the second set of extracted named entities from the first set of extracted named entities; 7. The method of claim 6, further comprising: (Appendix 8) verifying the trustworthiness of the first information associated with at least one of the document, the profile information, or the first set of named entities associated with the first candidate based on one or more public information databases; determining an empathy score associated with the first candidate is further based on the verification of trustworthiness; The method described in Appendix 6. (Appendix 9) the first set of characteristics includes at least one of a set of educational degrees, a set of certifications, years of work experience, a location, a set of languages ​​studied or known, a disability, a set of skills, an age, a gender, or a per capita income associated with the first candidate; The method described in Appendix 1. (Appendix 10) sending one or more queries to the one or more databases, the one or more queries being formed based on one or more combinations of the first set of features associated with the first candidates; extracting the second information related to the second set of features corresponding to the first set of features based on the one or more queries sent; 2. The method of claim 1, further comprising: (Appendix 11) the second set of characteristics includes at least one of an average per capita income, a set of educational degrees, a set of certifications, a number of job applications, a number of institutions, a number of graduate students, a number of jobs secured, or a set of languages ​​studied or known associated with the candidate population; The method described in Appendix 1. (Appendix 12) receiving a set of empathy scores for a set of documents in a training dataset for the neural network model based on user input from one or more raters; computing pairwise rankings among the received set of empathy scores for the set of documents; determining an optimal empathy score ranking for each of the document sets based on minimizing Kendall's tau distance across the calculated pairwise rankings; training the neural network model based on the determined optimal empathy score ranking for each of the document sets in the training dataset; and 2. The method of claim 1, further comprising: (Appendix 13) In response to being executed, the electronic device extracting first information regarding a first set of features associated with a first candidate from at least one of documents or profile information associated with the first candidate; extracting second information from one or more databases regarding a second set of features corresponding to the first set of features, the second set of features being associated with a population of candidates that includes at least one of the same set of demographic parameters as the demographic parameters of the first candidates; determining a third set of features based on corresponding feature differences from the first set of features associated with the first candidate and the second set of features associated with the population of candidates; applying a pre-trained neural network model to the determined third feature set, the neural network model being pre-trained to determine an empathy score for each of the candidate set based on the first feature set associated with the candidate set; determining a set of weights associated with the third feature set based on the application of the pre-trained neural network model; and determining an empathy score associated with the first candidate based on the determined set of weights; rendering the determined empathy score associated with the first candidate; and One or more non-transitory computer-readable storage media configured to store instructions for performing operations having: (Appendix 14) The operation is applying a regularization constraint to the pre-trained neural network model based on a criterion associated with one or more protected features from the first feature set associated with the first candidate; determining an empathy score associated with the first candidate is further based on applying the regularization constraint to the pre-trained neural network model. 14. The computer-readable storage medium of claim 13. (Appendix 15) The operation is sending one or more queries to the one or more databases, the one or more queries being formed based on one or more combinations of the first set of features associated with the first candidates; extracting the second information related to the second set of features corresponding to the first set of features based on the one or more queries sent; Further comprising: 14. The computer-readable storage medium of claim 13. (Appendix 16) The operation is verifying the trustworthiness of the first information associated with at least one of the document, the profile information, or a first set of named entities associated with the first candidate based on one or more public information databases; determining an empathy score associated with the first candidate is further based on the verification of trustworthiness; 14. The computer-readable storage medium of claim 13. (Appendix 17) a memory storing instructions; a processor, coupled to the memory, for executing the instructions to perform a process; and The process comprises: extracting first information regarding a first set of features associated with a first candidate from at least one of documents or profile information associated with the first candidate; extracting second information from one or more databases regarding a second set of features corresponding to the first set of features, the second set of features being associated with a population of candidates that includes at least one of the same set of demographic parameters as the demographic parameters of the first candidates; determining a third set of features based on corresponding feature differences from the first set of features associated with the first candidate and the second set of features associated with the population of candidates; applying a pre-trained neural network model to the determined third feature set, the neural network model being pre-trained to determine an empathy score for each of the candidate set based on the first feature set associated with the candidate set; determining a set of weights associated with the third feature set based on the application of the pre-trained neural network model; and determining an empathy score associated with the first candidate based on the determined set of weights; rendering the determined empathy score associated with the first candidate; and having Electronic devices. (Appendix 18) The process comprises: applying a regularization constraint to the pre-trained neural network model based on a criterion associated with one or more protected features from the first feature set associated with the first candidate; determining an empathy score associated with the first candidate is further based on applying the regularization constraint to the pre-trained neural network model. 18. The electronic device of claim 17. (Appendix 19) The process comprises: sending one or more queries to the one or more databases, the one or more queries being formed based on one or more combinations of the first set of features associated with the first candidates; extracting the second information related to the second set of features corresponding to the first set of features based on the one or more queries sent; Further comprising: 18. The electronic device of claim 17. (Appendix 20) The process comprises: verifying the trustworthiness of the first information associated with at least one of the document, the profile information, or a first set of named entities associated with the first candidate based on one or more public information databases; determining an empathy score associated with the first candidate is further based on the verification of trustworthiness; 18. The electronic device of claim 17. [Explanation of symbols]

[0121] 102 Electronic Devices 104 Candidate Information DB 106 Knowledge DB 108 Public Information DB 110 User End Devices 112 Communication Network 114 users 116 documents 118 Profile Information 120 Candidate Information 202 System 204 processors 206 memory 208 Persistent Data Storage 210 Input / Output (I / O) Devices 212 display screen 214 Network Interface 216 Neural Network Model

Claims

1. 1. A processor-implemented method comprising: extracting first information regarding a first set of features associated with a first candidate from at least one of documents or profile information associated with the first candidate; extracting second information from one or more databases regarding a second set of features corresponding to the first set of features, the second set of features being associated with a population of candidates that includes at least one of the same set of demographic parameters as the demographic parameters of the first candidates; determining a third set of features based on corresponding feature differences from the first set of features associated with the first candidate and the second set of features associated with the population of candidates; applying a pre-trained neural network model to the determined third set of features, the neural network model being pre-trained to determine an empathy score for each of the candidate sets based on the first set of features associated with the candidate set; determining a set of weights associated with the third feature set based on the application of the pre-trained neural network model; and determining an empathy score associated with the first candidate based on the determined set of weights; Rendering the determined empathy score associated with the first candidate; and A method having the following.

2. applying a regularization constraint to the pre-trained neural network model based on a criterion associated with one or more protected features from the first feature set associated with the first candidate; determining an empathy score associated with the first candidate is further based on applying the regularization constraint to the pre-trained neural network model. The method of claim 1.

3. the criterion corresponds to a counterfactual criterion that a first acceptance probability of the first candidate having a first value of a first protected feature of the one or more protected features is the same as a second acceptance probability of a second candidate having a second value of the first protected feature, based on a determination that other values ​​of the first set of features are the same. The method of claim 2.

4. the one or more protected characteristics include at least one of age, sex, or a disability associated with the first candidate; The method of claim 2.

5. the profile information associated with the first candidate includes information associated with at least one of a job cover letter, a set of recommendations, a source code repository, a project page, a professional networking web page, or a social networking web page associated with the first candidate; The method of claim 1.

6. extracting a first set of named entities from at least one of the documents or the profile information associated with the first candidate. The method of claim 1.

7. filtering the extracted first set of named entities to extract a second set of named entities; extracting the first set of features associated with the first candidate based on the exclusion of the second set of extracted named entities from the first set of extracted named entities; The method of claim 6 further comprising:

8. verifying the trustworthiness of the first information associated with at least one of the document, the profile information, or the first set of named entities associated with the first candidate based on one or more public information databases; determining an empathy score associated with the first candidate is further based on the verification of trustworthiness; The method of claim 6.

9. the first set of characteristics includes at least one of a set of educational degrees, a set of certifications, years of work experience, a location, a set of languages ​​studied or known, a physical disability, a set of skills, an age, a gender, or a per capita income associated with the first candidate; The method of claim 1.

10. sending one or more queries to the one or more databases, the one or more queries being formed based on one or more combinations of the first set of features associated with the first candidates; extracting the second information regarding the second set of features corresponding to the first set of features based on the submitted one or more queries; The method of claim 1 further comprising:

11. the second set of characteristics includes at least one of an average per capita income, a set of educational degrees, a set of certifications, a number of job applications, a number of institutions, a number of graduate students, a number of jobs secured, or a set of languages ​​studied or known associated with the candidate population; The method of claim 1.

12. receiving a set of empathy scores for a set of documents in a training dataset for the neural network model based on user input from one or more raters; computing pairwise rankings among the received set of empathy scores for the set of documents; determining an optimal empathy score ranking for each of the document sets based on minimizing Kendall's tau distance across the calculated pairwise rankings; training the neural network model based on the determined optimal empathy score ranking for each of the document sets in the training dataset; and The method of claim 1 further comprising:

13. In response to being executed, the electronic device extracting first information regarding a first set of features associated with a first candidate from at least one of documents or profile information associated with the first candidate; extracting second information from one or more databases regarding a second set of features corresponding to the first set of features, the second set of features being associated with a population of candidates that includes at least one of the same set of demographic parameters as the demographic parameters of the first candidates; determining a third set of features based on corresponding feature differences from the first set of features associated with the first candidate and the second set of features associated with the population of candidates; applying a pre-trained neural network model to the determined third set of features, the neural network model being pre-trained to determine an empathy score for each of the candidate sets based on the first set of features associated with the candidate set; determining a set of weights associated with the third feature set based on the application of the pre-trained neural network model; and determining an empathy score associated with the first candidate based on the determined set of weights; Rendering the determined empathy score associated with the first candidate; and One or more non-transitory computer-readable storage media configured to store instructions that cause the execution of operations having the following:

14. The operation is applying a regularization constraint to the pre-trained neural network model based on a criterion associated with one or more protected features from the first feature set associated with the first candidate; determining an empathy score associated with the first candidate is further based on applying the regularization constraint to the pre-trained neural network model. The computer-readable storage medium of claim 13.

15. The operation is sending one or more queries to the one or more databases, the one or more queries being formed based on one or more combinations of the first set of features associated with the first candidates; extracting the second information regarding the second set of features corresponding to the first set of features based on the submitted one or more queries; Further comprising: The computer-readable storage medium of claim 13.

16. The operation is verifying the trustworthiness of the first information associated with at least one of the document, the profile information, or a first set of named entities associated with the first candidate based on one or more public information databases; determining an empathy score associated with the first candidate is further based on the verification of trustworthiness; The computer-readable storage medium of claim 13.

17. a memory storing instructions; a processor, coupled to the memory, for executing the instructions to perform a process; and The process comprises: extracting first information regarding a first set of features associated with a first candidate from at least one of documents or profile information associated with the first candidate; extracting second information from one or more databases regarding a second set of features corresponding to the first set of features, the second set of features being associated with a population of candidates that includes at least one of the same set of demographic parameters as the demographic parameters of the first candidates; determining a third set of features based on corresponding feature differences from the first set of features associated with the first candidate and the second set of features associated with the population of candidates; applying a pre-trained neural network model to the determined third set of features, the neural network model being pre-trained to determine an empathy score for each of the candidate sets based on the first set of features associated with the candidate set; determining a set of weights associated with the third feature set based on the application of the pre-trained neural network model; and determining an empathy score associated with the first candidate based on the determined set of weights; Rendering the determined empathy score associated with the first candidate; and having Electronic devices.

18. The process comprises: applying a regularization constraint to the pre-trained neural network model based on a criterion associated with one or more protected features from the first feature set associated with the first candidate; determining an empathy score associated with the first candidate is further based on applying the regularization constraint to the pre-trained neural network model.

18. The electronic device of claim 17.

19. The process comprises: sending one or more queries to the one or more databases, the one or more queries being formed based on one or more combinations of the first set of features associated with the first candidates; extracting the second information regarding the second set of features corresponding to the first set of features based on the submitted one or more queries; Further comprising:

18. The electronic device of claim 17.

20. The process comprises: verifying the trustworthiness of the first information associated with at least one of the document, the profile information, or a first set of named entities associated with the first candidate based on one or more public information databases; determining an empathy score associated with the first candidate is further based on the verification of trustworthiness; 18. The electronic device of claim 17.

Images