Transformer architecture for machine learning model

Abstract

A transformer neural network may include a model. The model may receive a plurality of feature groups, the plurality of feature groups including a plurality of features. The model may process the feature groups, for example at a first layer of the transformer neural network to determine a relationship between the plurality of feature groups. The model may process the plurality of features, for example at a second layer of the transformer neural network to determine a relationship between the plurality of features. The plurality of feature groups may include feature groups associated with a candidate for a job. The feature may include features associated with the candidate for the job. The model may output an indication of the relationship between the plurality of feature groups and / or the plurality of features. For example, the indication may include a score associated with a suitability of the candidate for the job.

Description

[0001] TRANSFORMER ARCHITECTURE FOR MACHINE LEARNING MODEL

[0002] CROSS-REFERENCE TO PRIORITY INFORMATION

[0003] [1] This application claims the benefit of Indian Provisional Patent Application Number 202411069273, filed September 13, 2024. which is incorporated herein by reference in its entirety.

[0004] BACKGROUND

[0005] [2] Machine learning algorithms may employ artificial neural networks, which are loosely inspired from biological systems and / or include at least one hidden layer. Neural networks may be configured to transform an input sequence of data into an output sequence of data. A neural network, for example, a recurrent neural network, may be configured to transform an input sequence into an output sequence. The neural network can determine a context of the input sequence and process the input sequence to obtain a respective output sequence. The neural network may process each input as a hidden vector, at an encoder, and / or may process each vector, at a decoder, to obtain the output. Recently, neural networks have shown state-of-the-art performance in a variety of domains, e.g., speech, vision, natural language etc., and / or for various machine learning settings (e.g., supervised, unsupervised, and / or semi-supervised).

[0006] [3] A transformer is a type of sequence-to-sequence architecture. Similar to recurrent neural network models, transformers can process sequential data for tasks such as translation and text summarization, for example. However, unlike traditional models, such as recurrent neural networks (RNNs), which process sequences step by step in a specific order, transformers process the entire sequence at once. This is achieved through mechanisms like self-attention, which allows the model to focus on different parts of the input sequence simultaneously, capturing relationships between elements regardless of their position. This parallelism makes transformers faster and more efficient, especially when working with large datasets. Transformers have recently been used for solving various problems related to natural language and image processing. SUMMARY

[0007] [4] A transformer neural network may include a model. The model may receive a plurality of feature groups. For example, the plurality of feature groups may comprise a plurality of features. The model may process (e.g., each of) the plurality of feature groups, for example with respective first weights at one or more first layers of the transformer neural network to determine a relationship between the plurality of feature groups. For example, the model may determine the first weights using matrix multiplication across each embedded feature group. Additionally, or alternatively, the model may process (e.g.. each of) the plurality of features, for example with respective second weights at one or more second layers of the transformer neural network to determine a relationship between the plurality of features. For example, the model may determine the second weights using matrix multiplication across each embedded feature. The model may output an indication of the relationship between the plurality of feature groups and / or the relationship between the plurality of features.

[0008] [5] The model may determine the first weights and / or the second weights using a circle loss function, a triplet loss function and / or supervised contrastive loss function. In some examples the model may embed the (e.g, each) feature group of the plurality of feature groups with respective positional information. The plurality of feature groups may include feature groups associated with a candidate for a job. The feature may include features associated with the candidate for the job. The indication (e.g., output) may include a score associated with a suitability of the candidate for the job. In some examples, the plurality of feature groups may include one or more of a plurality of feature groups associated with a plurality of jobs, a first plurality of feature groups associated with a first plurality of candidates for one or more of the plurality of jobs, and / or a second plurality of feature groups associated with a second plurality of candidates for one or more of the plurality of jobs.

[0009] [6] A model may be trained with a transformer neural netw ork. The model may select a feature from a plurality of features, for example randomly. The model may remove data from the selected feature to generate synthetic data. Additionally, or alternatively, the model may add data to the selected feature to generate synthetic data. The model may input the synthetic data as training data, for example to the transformer neural network. The features may include features associated with a candidate for a job. The synthetic data may be associated with a real candidate for a job and / or a fictional (e.g, synthetic) candidate for a job. A model may be trained by selecting training data associated with a candidate for a job. The training data may include a plurality of classes for the candidate for the job. The model may embed (e.g., each of) the classes with a label. For example, the label may include one or more of positive, negative, or neutral. The training data may be input into the transformer neural netw ork model. In some examples the labels may be positive and negative. In some examples the labels may (e.g., only) be positive)

[0010] [7] A model may receive a plurality of feature groups in a transformer neural network. The plurality of feature groups may be associated with standardized job feature groups. The model may receive a plurality of features in the transformer neural network. The plurality of features may be associated with standardized job features. The model may process (e.g., each of) the plurality of feature groups at a first layer of the transformer neural netw ork to determine a relationship betw een the plurality of feature groups and / or process (e.g.. each of) the plurality of features at a second layer of the transformer neural netw ork to determine a relationship between the plurality of features.

[0011] [8] The model may output an indication of the relationship between the plurality of feature groups and / or the relationship betw een the plurality of features. The indication may be an indication of the relationship between a predetermined number of feature groups and / or the relationship betw een a predetermined number of features. For example, the predetermined number may be 50. The standardized job feature groups and / or the standardized job features comprise standardized feature groups or features of occupational data, for example from O*NET. In some examples, the plurality of features may include one or more of a plurality of features associated with a plurality of jobs, a first plurality of features associated with a first plurality of candidates for one or more of the plurality’ of jobs, and / or a second plurality of features associated with a second plurality of candidates for one or more of the plurality of jobs.

[0012] [9] A model may receive selected data associated with a candidate for a job. The model may receive selected job data associated with the job. The model may embed the data associated with the candidate for the job with a label. The model may input the data associated with the candidate for the job and / or the data associated with the job into a neural network. The model may determine a classification loss, for example based on the data associated with the candidate for the job and / or the data associated with the job. The model may determine a contrastive loss, for example based on the data associated with the candidate for the job and / or the data associated with the job. The model may determine a multi-objective loss, for example based on the classification loss and / or the contrastive loss.

[0010] A model may receive data associated with a candidate for a job and / or data associated with the job. The model may input the data associated with the candidate for the job and / or the data associated with the job into a neural network. The model may receive metadata from at least one of the data associated with the candidate for the job and / or the data associated with the job. The model may input the data associated with the candidate for the job. the data associated with the job, and / or the metadata from at least one of the data associated with the candidate for the job and / or the data associated with the job into a neural network, for example into the transformer neural network model.

[0013]

[0011] A model may receive data associated with a job and / or data associated with a candidate for the job. The model may determine at least one embedding for at least one feature of the data associated with the job. The model may determine the at least one embedding for the at least one feature of the data associated with the job based on the data associated with the job. In some examples, the model may determine the at least one embedding for the at least one feature of the data associated with the job not based on the data associated with the candidate for the job. The model may determine at least one embedding for at least one feature of the data associated with the candidate for the job. For example, the model may determine the at least one embedding for the at least one feature of the data associated with the candidate for the job based on the data associated with the candidate for the job. In some examples, the model may determine the at least one embedding for the at least one feature of the data associated with the candidate for the job not based on the data associated with the job.

[0014]

[0012] The model may determine, for example based on the at least one embedding for at least one feature of the data associated with the job and / or the at least one embedding for at least one feature of the data associated with the candidate for the job, at least one cross attention value associated with the at least one feature of the data associated with the candidate and / or at least one cross attention value associated with the job. The model may determine an indication of a suitability of the candidate for the job, for example based on the at least one cross attention value associated with the at least one feature of the data associated with the candidate and / or the at least one cross attention value associated with the job.

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016]

[0013] FIG. 1A is a schematic illustration of an example system environment that may implement an AI / ML model.

[0014] FIG. IB illustrates an example of a neural network.

[0017]

[0015] FIG. 1 C is a schematic illustration of an example system environment for training and implementing an AI / ML model that comprises a neural network.

[0018]

[0016] FIGs. 2A and 2B illustrate an example hiring progression.

[0019]

[0017] FIG. 3 illustrates an example of model training.

[0020]

[0018] FIG. 4A illustrates mean (e.g., average) pooling that may be used in a transformer architecture.

[0021]

[0019] FIG. 4B illustrates examples of transformer architectures with example attention pooling.

[0022]

[0020] FIG. 5 illustrates an example of a model architecture.

[0023]

[0021] FIG. 6 illustrates example input to an example model.

[0024]

[0022] FIG. 7 illustrates example transformers.

[0025] DETAILED DESCRIPTION

[0026]

[0023] Systems, methods, and / or apparatuses described herein may implement artificial intelligence (Al) and / or machine learning (ML) algorithms, for example in a human resources context. AI / ML may be deployed in a production environment, for example where an AI / ML model(s) is integrated in a recruitment system. The AI / ML model(s) may be configured to process jobs (e.g., job posts) and candidates (e.g, resumes), for example in real-time.

[0027]

[0024] FIG. 1A is a schematic illustration of an example system environment 101 that may implement an AI / ML 109 model. The AI / ML 109 model may include model data and one or more algorithms and / or functions configured to leam from input data 107 that is received to train the AI / ML 109 and / or generate an output 115. The input data 107 may be input in one or more formats, such as an image format, an audio format (e.g., spectrogram or other audio format), a tensor format (e.g., including single-dimensional or multi-dimensional arrays), and / or another data type capable of being input into the AI / ML 109 algorithms. The input data 107 may be the result of pre-processing 105 that may be performed on raw data 103, or the input data 107 may include the raw data 103 itself. Additionally, or alternatively, the input data 107 may include tokens, for example as discussed herein.

[0028]

[0025] The raw data 103 may include image data, text data, audio data, or another sequence of information, such as a sequence of network information related to a communication network, and / or other types of data. For example, pre-processing 105 may be selected based on (e.g., be specific to) the (e.g. type of) raw data. The pre-processing 105 may include one or more of cropping, rotation, and / or color conversion, for example for (e.g., raw) image data. The preprocessing may include stop word removal and / or lemmatization, for example for (e.g., raw) text data. The pre-processing 105 may include format changes or other types of processing in order to generate input data 107 in a format for being input into the AI / ML 109 algorithms. For example, image data, text data and / or audio data may be raw data 103 that may be pre-processed during pre-processing 105 to generate the input data 107 in a format configured to be received by the AI / ML 109 algorithm. The output 115 may be generated by the AI / ML 109 algorithm in one or more formats, such as a tensor, a text format (e.g.. a word, sentence, or other sequence of text), a numerical format (e.g., a prediction), an audio format, an image format, another data sequence format, or / another output format.

[0029]

[0026] AI / ML may be implemented as described herein using software and / or hardware. The AI / ML may be stored as computer-executable instructions on one or more computer-readable medium accessible by one or more processors for processing as described herein. Example AI / ML environments and / or libraries include Tensorflow, Torch, PyTorch, Matlab, Google Cloud Al and Automl, Amazon Sagemaker, Azure Machine Learning Studio, and / or Oracle Machine Learning. Additionally, or alternatively, a transformer, transformer architecture, and / or model as herein may be stored as computer-executable instructions on one or more computer- readable medium accessible by one or more processors for processing as described herein. The one or more computer-readable medium may include a non-transitory computer-readable medium.

[0030]

[0027] The AI / ML 109 may include one or more algorithms configured for unsupervised learning. Unsupervised learning may be implemented utilizing AI / ML 109 algorithms that learn from the input data 107 without being trained toward a particular target output. For example, during unsupervised learning the AI / ML 109 algorithms may receive unlabeled data as input data 107 and determine patterns or similarities in the input data 107 without additional intervention (e.g., updating parameters and / or hyperparameters). The AI / ML 109 algorithms that are configured for implementing unsupervised learning may include algorithms configured for identifying patterns, groupings, clusters, anomalies, and / or similarities or other associations in the input data 107. For example, the AI / ML may implement hierarchical clustering algorithms, k-means clustering algorithms, k nearest neighbors (K-NN) algorithms, anomaly detection algorithms, principal component analysis algorithms, and / or apriori algorithms.

[0028] An autoencoder may be a form of Al / ML 109 that may be implemented for unsupervised learning. The autoencoder may include an encoder configured to transform the input data 107 and / or a decoder that may recreate the input data from the data received by the encoder. The autoencoder may be implemented for processing image data and / or other forms of input data. The AI / ML 109 algorithms configured for unsupervised learning may be implemented on a single device or distributed across multiple devices, such that the output 115, or portions thereof, may be aggregated at one or more devices for being further processed and / or implemented in other downstream algorithms or processes, as may be further described herein.

[0031]

[0029] The AI / ML 109 may include one or more algorithms configured for supervised learning. Supervised learning may be implemented utilizing AI / ML 109 algorithms that are trained during a training process to determine a predictive model using known outcomes. The AI / ML 109 algorithms may be characterized by parameters and / or hyperparameters that may be trained during the training process. The parameters may include values derived during the training process. The parameters may include w eights, coefficients, and / or biases. The AI / ML 109 may also include hyperparameters. The hyperparameters may include values used to control the learning process. The hyperparameters may include a learning rate, a number of epochs, a batch size, a number of layers, a number of nodes in each layer, a number of kernels (e.g. , CNNs), a size of stride (e.g, CNNs), a size of kernels in a pooling layer (e.g., CNNs), and / or other hyperparameters. Some may use certain parameters and hyperparameters interchangeably.

[0032]

[0030] The AI / ML 109 may be trained during supervised learning by inputting training data to the AI / ML 109 algorithm and adjusting the parameters and / hyperparameters toward a known target output 115 while minimizing a loss or error in the output 115 generated by the AI / ML 109 algorithm. The raw' data 103 may include or be separated into training data, validation data, and / or test data for training, validation, and / or testing, respectively, the AI / ML 109 algorithms during supervised learning. The training data, validation data, and / or test data is pre-processed from the raw data 103 for being input into the AI / ML 109 algorithm. During supervised learning, the training data is labeled prior to being input into the AI / ML 109. The training data is labeled to teach the AI / ML 109 algorithm to leam from the labeled data and to test the accuracy of the AI / ML 109 for being implemented on unlabeled input data 107 during production / implementation of the AI / ML 109 algorithms, or similar AI / ML 109 algorithms utilizing similar parameters and / or hyperparameters.

[0033]

[0031] The training data may be used to fit the parameters of the AI / ML 109 model using optimization functions, such as a loss or error function. Often the training data includes pairs of input data 107 and a corresponding target output 115 to which the parameters may be trained to generate (e.g., within a threshold loss or error). The trained or fitted AI / ML 109 model may receive the validation data as input to evaluate the model fit on the training data set, while tuning the hyperparameters of the AI / ML 109 model. The AI / ML 109 model may receive the test data to evaluate generalization capabilities of the AI / ML 109 model. One or more of the training, validation, and / or testing may be performed during supervised learning for different types of AI / ML 109 models. Example AI / ML algorithms 109, trained using supervised learning, may be used in a human resources context. For example, the AI / ML model 109 may be used for matching a candidate and a job. The training data may include pairs of input data 107 and corresponding target output 115 based on past hiring decisions, for example of a particular company. AI / ML and / or AI / ML algorithm(s) as herein may include a transformer, transformer architecture, transformer neural network, and / or model.

[0034]

[0032] Supervised learning may be implemented for various types of AI / ML 109 algorithms, including algorithms that implement linear regression, logistic regression, neural networks (NNs), decision trees, Bayesian logics, random forests, and / or support vector machines (SVMs). NNs and Deep NNs (DNNs) are popular examples of algorithms utilized in AI / ML models that may be trained using supervised learning. However, the AI / ML 109 models may implement one or more NN and / or non-NN-based algorithms. Various examples of NNs include: perceptrons, multilayer perceptrons (MLPs), feed-forward NNs, fully-connected NNs, convolutional Neural Networks (CNNs), recurrent NNs (RNNs), long-short term memory' (LSTM) NNs, and / or residual NNs (ResNets). A perceptron is a NN that includes a function that multiplies its input by a learned weight coefficient to generate an output value. A feed-forward NN is a NN that receives input at one or more nodes of an input layer and moves information in a direction through one or more hidden layers to one or more nodes of an output layer. In a feed-forward NN, one or more nodes of a given layer may be connected to one or more nodes of another layer. A fully connected NN is a NN that includes an input layer, one or more hidden layers, and an output layer. In a fully connected NN, each node in a layer is connected to each node in another layer of the NN. An MLP is a fully connected class of feed-forward NNs. A CNN is a NN having one or more convolutional layers configured to perform a convolution.

[0035]

[0033] Various types of NNs may have elements that include one or more CNNs or convolutional layers, such as Generative Adversarial Networks (GANs). GANs may include conditional GANs (CGANs), cycle-consistent GANs (CycleGANs), StyleGANs, DiscoGANs, and / or IsGANs. A GAN may include a generator sub-model and a discriminator sub-model. The generator sub-model may be configured to receive input data and pass true and independently generated data to the discriminator sub-model. The discriminator sub-model may be configured to receive the true and independently generated data from the generator, discriminate the true and independently generated data, and provide feedback to the generator sub-model during training to improve the function of the generator sub-model in independently- generating an output based on a received input. The GAN is a popular model for generating data ty pes or data sequences, such as image data, audio data, and / or text, for example.

[0036]

[0034] An RNN is a NN that is recurrent in nature, as the nodes include feedback connections and an internal hidden state (e.g, memory) that allows output from nodes in the NN to affect subsequent input to the same nodes. LSTM NNs may be similar to RNNs in that the nodes have feedback connections and an internal hidden state (e.g, memory ). However, the LSTM NNs may include additional gates to allow the LSTM NNs to leam longer-term dependencies between sequences of data. A ResNet is a NN that may include skip connections to skip one or more layers of the NN. Some NNs include one or more attention layers or functions to enhance or focus on some portions of the input data, while diminishing or de-emphasizing other portions.

[0037]

[0035] Different types of NNs and / or layers may be implemented for processing different types of data and / or producing different types of output. For example, the NN may comprise of one or more convolutional layers, which may be popular for processing image data and / or audio data (e.g, spectrograms). Each convolutional layer may vary according to various convolutional layer parameters or hyperparameters, such as kernel size (e.g, field of view of the convolution), stride (e.g, step size of the kernel when traversing an image), padding (e.g, for processing image borders), and / or input and output size. The image being processed may include one or more dimensions (e.g, a line of pixels or a two-dimensional array of pixels). The pixels may be represented according to one or more values (e.g. one or more integer values representing color and / or intensity) that may be received by the convolutional layer. The kernel, which may also be referred to as a convolution matrix or mask, is a matrix used to extract and transform features from the input data being received. The kernel may be used for blurring, sharpening, edge detection. An example kernel size may include a 3x3, 5x5, 10x10, etc. matrix (e.g, in pixels for a 2D image). The stride may be the parameter used to identify the amount the kernel is moved over the image data. An example default stride is of a size of 1 or 2 within the matrix (e.g, in pixels for a 2D image). The padding may include the amount of data (e.g, in pixels for a 2D image) that is added to the boundaries of the image data when it is processed by the kernel. The kernel may be moved over the input image data (e.g, according to the stride length) and perform a dot product with the overlapping input region to obtain an activation value for the region. The output of each convolutional layer may be provided to a next layer of the NN or provided as an output (e.g., image data, feature map, etc.) of the NN itself with the updated features based on the convolution.

[0038]

[0036] The NN may include layers of a similar type (e.g., convolutional layers, feed-forward layers, fully -connected layers, etc.) and / or having a similar or different configuration (e.g, size, number of nodes, etc.) for each layer. The NN may also, or alternatively, include one or more layers having different types or different subsets of NNs that may be interconnected for training and / or implementation, as described herein. For example, a NN may include both convolutional layers and feed-forward or fully-connected layers.

[0039]

[0037] FIG. IB illustrates an example of a neural network 109a. The objective of training may be to apply the input 107a as training data and / or adjust one or more weights, indicated as w and x in FIG. IB (e.g., which may be referred to as neuron weights and / or link weights), such that the output 115 from the neural network 109a approaches the desired target values which are associated with the input 107a values for the training data. In examples, a neural network may include three layers (e.g., as shown in FIG. IB). During the training, for given input, the difference between output and desired values may be computed and / or the difference may be used to update the one or more weights in the neural network. If a significant (e.g., large) difference between output and desired value(s) is observed, for example, one or more relatively significant (e.g., large) changes in one or more weights may be expected. A small difference (e.g., between output and desired value(s) may include one or more relatively small changes in one or more weights.

[0040]

[0038] Once the neural network 109a completes training, the difference between the output 115 and desired values may be below a threshold. The neural network 109 is applied after training by feeding input data 107 and / or by estimating or predicting the output 115 as the expected outcome for the associated input 107. The paired input data 107 and output data 115 are used for human resources decisions. The input data 107 may include candidates who progressed to an interview stage, for example paired with output data 115 indicating that the candidate is positive. Additionally, or alternatively, the input data 107 may include candidates who did not progress to the interview stage, for example paired with output data 115 indicating that the candidate is negative. The AI / ML model 109 may learn positive and / or negative candidates using the labelled positive and / or negative candidate data.

[0039] Training a neural network 109 may include identifying one or more of input for the neural network, the expected output associated with the input, and / or the actual output from the neural network against which the target values are compared. In some examples, a neural network model may include (e.g., be characterized by) one or more parameters and / or hyperparameters. The one or more parameters and / or hyperparameters may include one or more of the number of weights and / or the number of layers in the neural network.

[0041]

[0040] As used herein, the term deep learning may refer to a class of machine learning algorithms that employ artificial neural networks (e.g., deep neural networks (DNNs)) which were loosely inspired from biological systems and / or include at least one hidden layer. DNNs may be a special class of machine learning models inspired by the human brain where the input is linearly transformed and / or pass through a non-linear activation function one or more (e.g., multiple) times. DNNs may include one or more (e.g.. multiple) layers where one or more (e.g., each) layer includes linear transformation and / or a given non-linear activation function(s). The DNNs may be trained using the training data via a back-propagation algorithm. Recently, DNNs have shown state-of-the-art performance in variety of domains, e.g., speech, vision, natural language etc., and / or for various machine learning settings (e.g., supervised, unsupervised, and / or semi-supervised).

[0042]

[0041] FIG. 1C is a schematic illustration of an example system environment 101a for training and implementing an AI / ML 109 model that comprises an NN 109a. However, other types of AI / ML models (e.g.. including NNs and / or non-NN models) may be similarly trained and / or implemented. The NN 109a may be trained and / or implemented on one or more devices to determine and / or update parameters and / or hyperparameters 117 of the NN 109a. Raw data 103a may be generated from one or more sources.

[0043]

[0042] The NN 109a may include one or more layers 111. The configuration of the NN 109a and / or the layers 111 may be based on the parameters and / or hyperparameters 117. As described herein, the parameters may include weights, or coefficients, and / or biases for the nodes or functions in the layers 111. The hyperparameters may include a learning rate, a number of epochs, a batch size, a number of layers, a number of nodes in each layer, a number of kernels (e.g., CNNs), a size of stride (e.g., CNNs), a size of kernels in a pooling layer (e.g., CNNs), and / or other hyperparameters. As described herein, the NN 109a may include a feed forward NN, a fully connected NN a CNN, a GAN, an RNN, a ResNet, and / or one or more other types of NNs. The NN 109a may be comprised of one or more different types of NNs or different layers for different types of NNs. For example, the NN 109a may include one or more individual layers having one or more configurations.

[0044]

[0043] Input data 107a may include standardized training data, for example for human resources use across multiple customers (e.g., employers), applicant tracking systems, and / or jobs. As a model may be used by multiple customers (e.g., employers), the model is be trained using standardized training data, for example for a particular occupation and / or field. For example, multiple applicant tracking systems may be integrated into the model input data 107a which may result in unique input data 107a. Each applicant tracking system may be configured (e.g.. uniquely) by a customer (e.g.. employer).

[0045]

[0044] Training data may be labeled, for example by labeling classes. For example, classes may be labelled positive, neutral, or negative. Positive may represent an acceptable candidate (e.g, for an interview), and negative may represent an unacceptable candidate. Neutral may represent that the class is irrelevant to the model. In some examples, the training data may be labeled using an automated model that classifies the customer (e.g, employer) data. The customer (e.g., employer) data may include a hiring progression. Data and information as herein may be used interchangeably.

[0046]

[0045] FIGs. 2A and 2B illustrate example hiring progressions 200, 220, 240. Hiring progression 200 includes eight stages, which may be labelled for training. One or more stage may be labeled, for example based on the hiring progression 200 decision(s). The one or more labeled stages may be input as training data, for example to a transformer architecture (e.g., model). For example, the hiring progression may include a lead 202 (e.g., candidate lead). At 204 the candidate may be rejected. If the candidate is rejected for example, a rejected applicant label 204 (e.g. negative label) may be added.

[0047]

[0046] The hiring progression 200 may include an applicant 206. The applicant 206 may be prescreen rejected 208. If the applicant 206 is rejected for example, a pre-screen rejected label 208 (e.g., negative label 219) may be added. The applicant 206 may be screened, for example if a pre-screen is not rejected. If the applicant 206 is not rejected for example, a screen label 210 (e.g, neutral label 218) may be added. The applicant 206 may be rejected for an intend ew at 212, for example after the screen 210. If the applicant 206 is rejected for interview for example, a rejected interview 212 (e.g, negative label 219) may be added.

[0048]

[0047] The applicant 206 may be selected for interview at 214, for example after the screen 210. If the applicant 206 is selected for interview for example, an interview label 214 (e.g., positive label 218) may be added. After the applicant 206 interview (e.g, at 214) for example, there may be a post interview at 216. For example, a post interview label 216 e.g., positive label 218) may be added.

[0049]

[0048] A hiring progression 220 may include a new submission 222 (e.g, from a candidate). At 224 a recruiter may review the submission 222. At 226 a manager may review the submission 222, for example if the recruiter review 224 is positive. At 228 the candidate (e.g., of the submission 222) may be interviewed, for example if the manager review 226 is positive. At 230 the job (e.g, associated with the submission 222) may be closed, for example if the candidate was hired, if the job was cancelled, and / or if another candidate was hired. One or more stages of the hiring progression 220 may be labeled, for example for training. The one or more labeled stages may be input as training data, for example to a transformer architecture (e.g., model). For example, the submission 222, the recruiter review 224, the manager review 226, the interview 228, and / or the closing of the job 230 may be labeled with a positive label 232.

[0050]

[0049] A hiring progression 240 may include a new submission 242 (e.g., from a candidate). At 244 a recruiter may review the submission 242. At 246 the job (e.g., associated with the submission 242) may be closed, for example if the candidate was hired, if the job was cancelled, and / or if another candidate was hired. One or more stages of the hiring progression 240 may be labeled, for example for training. The one or more labeled stages may be input as training data, for example to a transformer architecture (e.g. , model). For example, the submission 242, the recruiter review 244, and / or the closing of the job 246 may be labeled with a neutral label 248.

[0051]

[0050] A user may add a label to training data, for example positive, negative, or neutral. For example, a user may add a label to training data before the training data is input to a model. A model (e.g., neural network) may receive the labeled data and / or may predict a label (e.g, of other input data) based on training. Additionally, or alternatively, a model, for example of a transformer neural network, may determine a label of data based on training. The class of data as herein may be referred to by a label. For example, data of a positive class may include a positive label, data of a negative class may include a negative label, and / or data of a neutral class may include a neutral label.

[0052]

[0051] While transformers are known, a transformer architecture has not been used in a human resources context, for example to make applicant decisions. Since a transformer may contain self attention modules that may be positionally invariant and / or input in a human resources context is likely to have an ordered structure (e.g, a job title and / or description) which may be different from the structure of sequential text, positional embeddings may be useful in a human resources context. For example, a two layer approach may be used to converge features, such that a first layer may determine relative importance between feature groups, and a second layer may determine relative importance between different features within a feature group.

[0053]

[0052] A feature group may include one or more features. A feature group may include a feature group of skill features, a feature group of title features, a feature group of experience features, a feature group of geographic features, and / or a feature group of education features. Skill features may include a plurality of features, for example Python and Deep Learning. For example, skill features (e.g., of a candidate) may be received from a resume of the candidate. Additionally, or alternatively, skill features (e.g., of a job) may be received from a job posting. A feature group and / or a feature may be associated with a candidate, for example received from resume input data. A feature group may be associated with a job, for example received from a job posting.

[0054]

[0053] An example skill feature group may include skill features of the candidate and / or of the job (e.g. , job posting). Example features of a feature group of title features may include a previous title, a current title, a desired title, a similar title, and / or a (e.g. , posted) job title. Example features of a feature group of experience features may include previous experience (e.g, previous job), previous duties, previous accomplishments, current experience, current duties, current accomplishments, desired experience, desired duties, and / or desired accomplishments. Example features of a feature group of geographic features may include an address, a city7, a town, a state, a country7, a continent, and / or a distance from another location (e.g., an office). Example features of a feature group of education features may include a school name, year(s) associated with a school, a major, a course, a certification, and / or a license. A (e.g., any) feature group and / or feature may be associated with (e.g., read from) a candidate (e.g., resume) and / or ajob (e.g., job posting).

[0055]

[0054] A feature group and / or feature may include a standardized feature group and / or standardized feature. There may not be enough (e.g., useful) data, for example for some jobs. Additionally, or alternatively, there may be many terms used in data, for example a resume and / or job posting, with a similar meaning. A model may use standardized data (e.g., feature groups and / or features) to avoid unnecessary processing and / or inaccurate processing, for example of similar terms. For example, a model may receive a resume titled data scientist and another resume titled machine learning engineer. Some models may determine that these titles are significantly different, for example inaccurately. By using standardized data, inaccurate determinations may be avoided. For example, standardized data may be used for a job posting. This may additionally, or alternatively, avoid time of creating a job post. For many jobs a standardized description and / or requirements should result in a qualified candidate hire. Therefore, a model may use standardized data for the candidate data (e.g., for training) and / or for job data (e.g, for training and / or scoring). As standardized data may already be available, computing resources and / or time may be saved. For example, standardized data may include feature groups and / or features.

[0056]

[0055] For example, a feature group may include a standardized job feature group and / or a feature may include a standardized job feature. A standardized job feature group and / or a standardized job feature may include a standardized job feature group and / or a standardized job feature from occupational data, for example an occupational data database. For example, the occupational data may be from O*NET and / or European skills, competences, and occupations (ESCO). A standardized feature group and / or a standardized feature may include a standardized label. For example, a model may determine that a standardized feature group and / or a standardized feature includes a standardized label and / or may train a neural network based on the standardized feature group and / or a standardized feature with a standardized label. Additionally, or alternatively, a model may determine standardized data from a hiring progression, for example 200, 220, and / or 240.

[0057]

[0056] A model may include any number of (e.g., unique) hiring progressions (e.g., 200, 220, 240). For example, a model may include 156k unique hiring progressions across many clients such that a unique data asset is input into the model. The transformer based model may be trained to classify each customer (e.g., employer) hiring progression, for example using the standard hiring progression. For example, a hiring progression may be positive when a candidate is selected to interview (e.g., at 214). A hiring progression may be negative when a candidate is not selected to interview (e.g., at 212). In some examples a hiring progression (e.g., 200, 220, 240) may be input as raw data 103a. The raw data 103a may be input into a sentence transformer (e.g, neural network 109a) trained using supervised contrastive loss, for example such that an embedding space where unstandardized hiring statuses of a standard hiring status are close to each other. The transformer (e.g., neural network 109a) may additionally, or alternatively be trained using ordinal relationships among the classes. An embedding space of possible hiring progressions may be created, for example such that the embeddings may be used for predicting the classification of standard hiring statuses.

[0058]

[0057] Training data 107a may include synthetic data, for example to compensate for a low quantity’ of real-world examples. In some examples, training data 107a may be created using one or more of real data and / or synthetic data. For example, training data 107a may be created using real data and synthetic data. A synthetic training example of a candidate profile may be created by using a real candidate profile as a reference. A synthetic training candidate profile may be a relatively more positive or more negative example than the reference. A synthetic training example may be made from a real example by changing features of the real example. For example, features may be added, modified, and / or removed from or to the real example. For example, a synthetic example may include more skills, fewer skills, and / or different skills than the real example. Other features of an example, including job titles, education history7, experience, and / or proximity' may additionally, or alternatively be altered by adding, modifying and / or removing them. Any of adding, modifying, and / or removing operations may make a synthetic example positive or negative. Synthetic data may allow input of more training data 107a than otherwise may be available.

[0059]

[0058] Synthetic examples may be generated randomly. For example, random features may be removed and / or added and / or modified. Additionally, or alternatively, synthetic data may be generated completely (e.g, to create a new candidate resume). In some examples, a feature may be selected (e.g., by a user) for removal and / or addition. Synthetic data may be generated to be plausible (e.g, a plausible resume). A model may generate synthetic data by randomly changing feature groups and / or features, for example of a candidate and / or ajob. Randomness may increase the likelihood that synthetic data (e.g., a synthetic candidate) is a realistic candidate. Training using realistic synthetic data (e.g, candidates) may increase the accuracy of the model. For example, an unrealistic perfect candidate for a position used as synthetic data for training may skew a model to lower real candidate scores and / or (e.g, therefore) decrease accuracy of the model. Decreased accuracy of a model may demand more computing resources and / or time. Therefore, synthetic data and / or randomized synthetic data is desirable to increase model accuracy, save time, and / or avoid unnecessary use of computing resources.

[0060]

[0059] As an illustrative example, a real job candidate, R. may be considered for ajob. J. The model may predict the fit of the applicant to the job fit (R, J). The model may be trained to estimate a fit function. For example, the model may be trained with real data and / or synthetic data. Synthetic data may allow input of more training data 107a than otherwise available. For example, there may not be enough data available, especially for certain jobs (e.g, data scientist), to train a model to be accurate for predicting a candidate fit to ajob. Therefore, by generating synthetic job and / or candidate data, a model may be trained to make accurate predictions. Additionally, or alternatively, by creating positive (e.g., more positive) and / or negative (e.g., more negative) data, a model may be trained with increased accuracy since very' positive and / or very negative data may be used in training. For example, avoiding neutral data may’ increase processing speed and avoid unnecessary use of computing resources. Additionally, or alternatively, neutral data may not add much to model accuracy. Therefore, selecting and / or generating data (e.g, synthetic data) that is positive and / or negative may result in a more accurate model and better predictions. For example, only positive data may be selected and / or generated for training (e.g, as herein). Additionally, or alternatively, only negative data may be selected and / or generated for training (e.g, as herein).

[0061]

[0060] The synthetic data may include positive and / or negative data associated with one or more feature groups and / or features (e.g., as herein). Additionally, or alternatively, synthetic data may include synthetic data of a candidate and / or a job. For example, synthetic data may include a synthetic feature group and / or synthetic feature.

[0062]

[0061] A synthetic feature group may include one or more synthetic features. A synthetic feature group may include a synthetic feature group of synthetic skill features, a synthetic feature group of synthetic title features, a synthetic feature group of synthetic experience features, a synthetic feature group of synthetic geographic features, and / or a synthetic feature group of synthetic education features. Synthetic skill features may include a plurality of synthetic features, for example Python and Deep Learning. For example, synthetic skill features (e.g, of a candidate) may be generated by reading a resume and increasing or decreasing the positivity of the skill features of the candidate (e g., as herein). Additionally, or alternatively, skill features (e.g., of a job) may be read from a job posting and synthetic skill features of the job may be generated by increasing or decreasing the positivity of the skill features of the job (e.g, as herein). A synthetic feature group and / or a synthetic feature may be associated with a candidate, for example generated from resume input data. A synthetic feature group may be associated with a job, for example generated from a job posting.

[0063]

[0062] An example synthetic skill feature group may include synthetic skill features of the candidate and / or of the job (e.g, job posting). Example synthetic features of a synthetic feature group of synthetic title features may include a previous title, a current title, a desired title, a similar title, and / or a (e.g, posted) job title. Example synthetic features of a feature group of experience features may include previous experience (e.g. previous job), previous duties, previous accomplishments, current experience, current duties, current accomplishments, desired experience, desired duties, and / or desired accomplishments. Example synthetic features of a feature group of geographic features may include an address, a city, a town, a state, a countiy. a continent, and / or a distance from another location (e.g. an office). Example synthetic features of a feature group of education features may include a school name, year(s) associated with a school, a major, a course, a certification, and / or a license. A (e.g, any) synthetic feature group and / or synthetic feature may be associated with (e.g, generated from) a candidate (e.g, resume) and / or a job (e.g., job posting).

[0064]

[0063] FIG. 3 illustrates an example of model training 300. A model (e.g, in a transformer architecture) may be trained using positive information, negative information, and / or neutral information. For example, data labeled positive (e.g., 218, 232), negative (e.g, 219), and / or neutral (e.g, 248). However, in the example of model training 300, neutral information is not used for training. Neutral information (e.g, labels) may decrease model accuracy, increase computation speed, and / or increase computing resources needed for training. Therefore, by not using neutral information (e.g, labels) for training, model accuracy may be increased, computation speed may be increased, and / or computing resources needed for training may be decreased. Additionally, or alternatively, negative information (e.g, labels) may not be used for training in some examples. For example, only positive information (e.g, labels) may be used for training in some examples. By using only positive information (e.g, labels) for training, model accuracy may be further increased, computation speed may be further increased, and / or computing resources needed for training may be further decreased.

[0065]

[0064] In the example of model training 300, job data 302 may be input into a transformer architecture 306 (e.g., Fitformer). Additionally, or alternatively, candidate data 304 may be input into the transformer architecture 306, for example a model in the transformer architecture 306. The candidate data 304 (e.g. labeled information) may include positive candidate information (e.g, positive labels).

[0066]

[0065] The transformer architecture 306 (e.g, model) may output a loss function 310. For example, the loss function 310 may include classification loss. A classification loss (e.g. function) may indicate (e.g, quantify) error between a model prediction and the label (e.g., of the candidate information 304 input). Additionally, or alternatively, the transformer architecture 306 (e.g, model) may output job and / or candidate embeddings 308. The candidate embeddings may be used (e.g, in a model) to evaluate a candidate, for example by computing a candidate embeddings and comparing the candidate embeddings to the job embeddings. A candidate with candidate embeddings closer to the job embeddings may have a higher suitability’ score and / or classification.

[0067]

[0066] The model training 300 may determine contrastive loss 312. For example, the model training may determine contrastive loss 312 based on the job and / or candidate embeddings 308. The contrastive loss 312 may indicate a similarity (e.g, distance) between the job and candidate embeddings 308. A model may use the contrastive loss 312 to determine a candidate’s suitability (e.g., score) for a job.

[0068]

[0067] The model training 300 may determine scalarized multi objective loss 314. For example, the model training 300 may determine scalarized multi objective loss 314 based on the contrastive loss 312 and / or the classification loss 310. The model training 300 may weigh each of the contrastive loss 312 and the classification loss 310 to obtain the scalarized multi objective loss 314 (e.g., function). For example, weights may be selected such that the contrastive loss 312 is weighted more than the classification loss 310. A weight may be determined by a user (e.g, subject matter expert). A weight may additionally, or alternatively, be determined by a model, for example to optimize performance metric(s) on data (e.g, a dataset). An example dataset may include a validation dataset. Performance metrics may include metrics of machine learning, for example confusion matrix metrics. F-score, and / or loss.

[0069]

[0068] In the example of model training 300, job data 316 may be input into a transformer architecture 320 (e.g., Fitformer), for example a different transformer architecture 320 than transformer architecture 306. Additionally, or alternatively, candidate data 318 may be input into the transformer architecture 320, for example a model in the transformer architecture 320. The candidate data 318 (e.g., labeled information) may include negative candidate information (e.g., negative labels).

[0070]

[0069] The transformer architecture 320 (e.g, model) may output a loss function 324. For example, the loss function 324 may include classification loss. A classification loss (e.g., function) may indicate (e.g., quantify) error between a model prediction and the label (e.g., of the candidate information 318 input). Additionally, or alternatively, the transformer architecture 320 (e.g. model) may output job and / or candidate embeddings 322. The candidate embeddings may be used (e.g, in a model) to evaluate a candidate, for example by computing a candidate embeddings and comparing the candidate embeddings to the job embeddings. A candidate with candidate embeddings further from the job embeddings may have a lower suitability score and / or classification.

[0071]

[0070] The model training 300 may determine contrastive loss 312. For example, the model training may determine contrastive loss 312 based on the job and / or candidate embeddings 322. The contrastive loss 312 may indicate a dissimilarity (e.g., distance) between the job and candidate embeddings 322. A model may use the contrastive loss 324 to determine a candidate’s suitability (e.g., score) for a job.

[0071] The model training 300 may determine scalarized multi objective loss 314 based on the contrastive loss 312 and / or the classification loss 324. The model training 300 may weigh each of the contrastive loss 312 and the classification loss 324 to obtain the scalarized multi objective loss 314 (e.g.. function). For example, weights may be selected such that the contrastive loss 312 is weighted more than the classification loss 310. Additionally, or alternatively, the model training 300 may determine the scalarized multi objective loss 314 based on the contrastive loss 312, the classification loss 310, and / or the classification loss 324.

[0072]

[0072] A loss function 113 may additionally, or alternatively, include circle loss, triplet loss, and / or contrastive loss. Triplet loss may include a reference input matched to a positive input and a negative input, where the positive reference input may be a matching input and the negative reference input may be a non-matching input. A distance from the reference input to the positive input may be minimized and / or a distance from the reference input to the negative input may be maximized. For example, the reference may refer to a job. Positive may refer to (e.g., be associated with) any e.g., all) candidates who could (e.g., did) make the interview and / or beyond. Negative may refer to (e.g, be associated with) any (e.g., all) candidates who could not (e.g., did not) make the interview stage.

[0073]

[0073] The model may use circle loss, triplet loss, and / or contrastive loss, and / or the synthetic data may include a synthetic candidate profile that is assumed to be more positive or more negative than a real applicant to the same job. With a real applicant R to job, J synthetic negative applicants S- to the same position may be created, where fit(S-. J) < fit(R, J). Fit(S-, J) may likely be less than fit(R, J). Additionally, or alternatively, with the real applicant R to job J, synthetic positive applicants S+ to the same position may be created, where fit(S+, J) > fit(R, J). Fit(S+, J) will likely be greater than fit(R, J). The real applicant profile R may be intentionally degraded to create S-. The degrading procedure may be performed on any feature of R. For example:

[0074] If applicant R has some set of skills, then: Degradable skills <- skills(R) Degraded_skills <- select_skill_subset(Degradable_skills) skills(S-) <- Degraded_skills select_skill_subset may choose one or more skills from Degradable_skills. The select_skill_subset may choose using random weighted or unw eighted sampling or by any other method.

[0075] If applicant R has some set of previous titles, then: Degradable titles <- titles(R) Degraded_titles <- select_title_subset(Degradable_titles, n) titles(S-) <- Degraded titles select_title_subset removes one or more job titles from Degradable_titles. The select_title_subset may choose using random weighted or unweighted sampling or by any other method.

[0076]

[0074] Similarly, one or more other features (e.g., geographic distance, experience, and / or education) may be degraded. For example, geographic distance may be degraded by creating a synthetic negative profile to have a location further away from the job than R’s location. Experience may be degraded by creating a synthetic negative profile to have less or more experience relative to R’s experience. Education may be degraded by creating a synthetic negative profile to be underqualified or overqualified relative to R’s level of education.

[0077]

[0075] Additionally, or alternatively training data 107a may include one or more synthetic applicants. For example, a synthetic applicant S+ to the same job may be created, where fit(S+, J) > fit(R,J). The synthetic applicant S+ may be created as described herein. For example, the synthetic applicant S+ by may be created by upgrading R to be closer to some labeled example, U that is assumed (e.g. known) to be a strong fit for the position. The labeled example U may be identified. For example, a customer (e.g., employer) may indicate that there is an ideal candidate for some position. The labeled example U may alternatively, or additionally be an incumbent in a position (e.g., an employee currently in the position, previously held the position, or was previously offered the position). In some examples, data quality measures may be used to ensure that the labeled example U is sufficiently data rich and high quality.

[0078]

[0076] The model may sample one or more features F (e.g.. titles, skills, experience, geographic distance, and / or education) from the labeled example U profile. Features may be w eighted as described herein. For example, the model may give more weight to features that are relevant to the job J. The model may upgrade the real candidate R value for feature F to be closer to the labeled example U value for feature F. For example:

[0079] If F is skills, then skills(S+) <- skills(R) + sample(skills(U)) If F is titles, then titles(S+) <- titles(R) + sample(titles(U)) If F is experience, then experience(S+) <- average(experience(R), expert ence(U)), where averageQ is any weighted or unweighted averaging function

[0080] Similarly, one or more other features (e.g., geographic distance, experience, and / or education) may be upgraded.

[0077] During the training process, the training data 107a may be input into the NN 109a and may be used to learn the parameters and / or tune the hyperparameters 117. The training may be performed by initializing parameters and / or hyperparameters of the NN 109a, generating and / or accessing the training data 107a, inputting the training data 107a into the NN 109a. calculating the error or loss from the output of the NN 109a to a target output 115a via a loss function 113 (e.g., utilizing gradient descent and / or associated back propagation), and / or updating the parameters and / or hyperparameters 117.

[0081]

[0078] The loss function 113 may be configured such that standardization is preserved in the labels. For example a prediction of a rejected interview may be weighted less than a pre-screen rejected prediction, for example when predicting for a true label of an interview. Similarly, a rejected applicant may be weighted less than an interview prediction, for example when predicting for a true label of applicant.

[0082]

[0079] The loss function 113 may include circle loss, triplet loss, and / or supervised contrastive loss. For example, the circle loss, triplet loss, and / or supervised contrastive loss may be customized. The loss function 113 may be implemented using backpropagati on-based gradient updates and / or gradient descent techniques, such as Stochastic Gradient Descent (SGD), synchronous SGD, asynchronous SGD, batch gradient descent, and / or mini-batch gradient descent. Examples of loss or error functions may include functions for determining a squared- error loss, a mean squared error (MSE) loss, a mean absolute error loss, a mean absolute percentage error loss, a mean squared logarithmic error loss, a pixel-based loss, a pixel-wise loss, a cross-entropy loss, a log loss, and / or a fiducial-based loss. The loss functions may be implemented in accordance one or more quality metrics, such as a Signal to Noise Ratio (SNR) metric or another signal or image quality metric.

[0083]

[0080] The loss function 113 may be implemented as a supervised contrastive loss function. A supervised contrastive loss function may enhance the neural network 109a ability' to differentiate between positive and negative candidates. Unlike some classification losses, supervised contrastive loss may embed examples such that similar (e.g. positive) examples are closer in the embedding space, while dissimilar (e.g, negative) examples are farther apart, which may for example refine the model decision boundaries.

[0084]

[0081] A loss function 113 may be specific to the model. For example, in language modeling, loss function may be specific to language model. The classification task may be based on mask modeling (e.g., using a cross-entropy loss function) or modeling an entire sentence (e.g., using cross entropy loss). Circle loss, triplet loss, and / or supervised contrastive loss may be used, for example in image classification.

[0085]

[0082] The selection of one loss function over another loss function may be particular to a transformer architecture, for example a transformer architecture as herein. Additionally, or alternatively, the selection of the loss function may be particular to a usage, for example in a human resources context as disclosed herein. Cross entropy loss may be a supervised loss function. Contrastive loss may be a self supervised loss function. For example a supervised contrastive loss function may be supervised and / or may be trained in a self supervised way when combined with contrastive loss.

[0086]

[0083] The loss function 113 may include cross entropy loss. Cross entropy loss may be (e.g., heavily) affected by outliers and / or may not learn good patterns from imbalance data. In some examples, the loss function 113 may include contrastive loss. Contrastive loss may also be (e.g., heavily) affected by outliers and / or may not leam good patterns from imbalance data. However, contrastive loss may be less affected when compared with cross entropy loss. For example in an image classification task, there may be thousands of classes and very little data for each class available. Creating an embedding space where each class is well separable may be important despite minimal data. Contrastive losses may be used to create an embedding space where each class is well separable even when there is minimal data. For example in image and / or text classification, augmentation techniques may be used to generate (e.g.. many) more data points and / or (e.g., all of) the data may (e.g., then) be learned effectively by the model in a self supervised setting.

[0087]

[0084] Fitformer (e.g, a custom transformer model) may use contrastive loss to effectively treat each job and (e.g., positive) candidates as one (e.g., its own) class. Cross entropy may be used for classification problems, for example with neural networks. Cross entropy may be used to measure the difference between the true label distribution and the predicted probability distribution.

[0088] For example, for binary classification, an example formula is: where:

[0089] 1. ‘y ' is the true label (e.g., 0 or 1),

[0090] 2. ‘p ’ is the predicted probability (e.g., output of the model).

[0091] Where:

[0092] 1. ‘N’ is the number of classes,

[0093] 2. ‘y, ‘ is a binary indicator (e.g., 0 or 1) for whether class ‘i ' is the correct class,

[0094] 3. ‘pt ’ is the predicted probability for class ‘i

[0095]

[0085] The loss function 113 may be a circle loss function, for example as described in Sun, Y., Cheng, C., Zhang, Y ., Zhang, C., Zheng, L., Wang, Z., & Wei, Y. (2020). Circle loss: A unified perspective of pair similarity optimization. 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPRf https: / / doi.org / 10.1109 / cvpr42600.2020.00643, which is hereby incorporated by reference. Circle loss may be used for intrinsic hard mining, which may for example make manually mining for hard triplets difficult. Circle loss may dynamically adjust parameters (e.g., updates), for example based on distances from an anchor. A dot product or cosine similarity may be used as a distance metric, for example instead of Euclidean distance. For circle loss a sample formula may be:

[0096] Where:

[0097] 1. ‘SP‘ ’ is the similarity between anchor and ‘ith’positive candidate

[0098] 2. ‘AP' is the with-in class margin or positive class margin

[0099] 3. ‘SA ’ is the similarity between anchor and J ‘th’ negative candidate

[0100] 4. ‘An ’ is the between class margin or negative class margin

[0101] 5. ‘dP' is the positive class weighting factor which is non negative

[0102] 6. ‘oln ’ is the negative class weighting factor which is non negative

[0103] 7. ‘y’ is the scaling factor

[0104]

[0086] An optimizer may be implemented along with the loss function 113. The optimizer may be an algorithm or function that is configured to adapt attributes of the NN 109a, such as a learning rate and / or weights, to improve the accuracy of the NN 109a and / or reduce the loss or error. The optimizer may be implemented to update the parameters and / or hyperparameters 117 of the NN 109a. Weights may be adjusted by attention mechanisms, for example in a transformer architecture. Attention mechanisms may dynamically learn and / or adjust the weights of the NN 109a, for example based on the importance of different input data (e.g., features). The model may focus on the most relevant parts of the job and candidate data such that the overall accuracy of matching a candidate and a job is improved.

[0105]

[0087] The AI / ML model may utilize attention mechanisms within an transformer architecture, for example to leam the importance (e.g., what weight to apply) of features. The model may be a foundation model, for example for determining whether a candidate will proceed to the interview stage. The attention mechanisms may apply different weights to corresponding features (e.g.. of a candidate and / or ajob). For example, attention mechanisms may apply weights using selfattention such that the weights are determined automatically or by a user.

[0106]

[0088] Automatic weights may be used to determine a relationship between (e.g., different) features, for example for learning Fitformer. The automatic weights may be used in the transformer architecture. A two layer approach may be used to converge features. A first layer (e.g., of the NN) may determine a relative importance of skills within a (e.g., each) feature group. For example, the first layer may determine whether a first skill is more important than a second skill. A second layer may determine which features contribute more to the placement of a candidate between feature groups, for example job title v. skills. The second layer may determine which features contribute to overall embedding. This layer may use attention, for example as described in Bahdanau, D., Cho, K., & Bengio, Y. (2016, May 19). Neural machine translation by jointly learning to align and translate. arXiv.org. https: / / arxiv.org / abs / 1409.0473. which is hereby incorporated by reference. Additionally, or alternatively, the layer may use selfattention, for example as described in Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2023, August 2). Attention is all you need. arXiv.org. https: / / arxiv.org / abs / 1706.03762. which is hereby incorporated by reference.

[0107]

[0089] The output of an attention block in the transformer architecture may be of the same dimension as that of the input. However, a model may output a (e.g., single) representation descriptive of a candidate or ajob with accuracy. To achieve this output, (e.g, all of) the vectors may be averaged. The averaged vectors may be an average embedding such that the average embedding may be used as a document vector. Alternatively, or additionally to averaging, attention pooling may be used. Attention pooling may include learning weights with which the final embedding is calculated. Attention pooling may be used to select the (e.g, best) features that contribute to the overall embedding.

[0090] A transformer architecture, transformer, and / or neural network as herein may include a transformer neural network. A transformer neural network may include one or more heads, attention blocks, attention layers, and / or attention mechanisms. For example, an attention mechanism herein may include an attention layer and / or attention block. An attention mechanism may be configured to weigh a feature and / or feature group, for example by learning the importance of a feature or feature group. For example, an attention mechanism may apply weights using self-attention such that the weights are determined automatically or by a user.

[0108]

[0091] A head of a transformer neural network may be configured to output a decision, for example a fit of a candidate to ajob. The transformer neural network may include multiple heads, for example for multi-objective training. For example, the multiple heads may include a classification head, an embedding head, and / or a contrastive learning head. The classification head may be configured to output a value related to a decision (e.g, fit of a candidate to a job). The embedding head may be configured to process embeddings of input data, for example candidate data and / or job data. The embedding head may process embeddings, for example a subset of embeddings. The embedding head may process the embeddings to obtain crossattention. For example, an embedding head may obtain a different cross-attention than another embedding head by processing a different subset of embeddings.

[0109]

[0092] Some feature groups and / or features may be more important for determining a score, for example a fit of a candidate to ajob. Determining which feature groups and / or features are more important may result in a more accurate score determination. Additionally, or alternatively, determining which feature groups and / or features are more important may avoid unnecessary use of computing power and time, for example by avoiding repeated training, repeated scoring, and / or inaccurate scoring. Which feature groups and / or features may be more important may be user defined, read from the input data, and / or determined based on volume or positivity of training. For example, a model may read input data and determine importance of feature groups and / or features based on context (e.g., from a resume and / or job posting). For example, a model may determine that more prominent (e.g., earlier, larger, and / or repeated) feature groups and / or features are more important.

[0110]

[0093] Additionally, or alternatively, a model may determine which feature groups and / or features are more important based on a volume of feature groups and / or features in the input and / or training data (e.g. , from a resume and / or a job posting). A model may determine which feature groups and / or features are more important based on positivity of feature groups and / or features in the input and / or training data (e.g., from a resume and / or ajob posting). The model may determine weights for the associated feature groups and / or features based on the determined importance of the feature groups and / or features, for example equitably.

[0111]

[0094] Output as herein may include a fit (e.g, of a candidate to a job), a score (e.g, of a candidate to a job), a loss (e.g. , loss function), and / or training data. A relationship as herein may include the fit (e.g, of a candidate to a job) and / or the score (e.g, of a candidate to ajob). The relationship between a feature group of the candidate and a feature group of the job may include a difference between the feature group of the candidate and the feature group of the job. For example, a model may determine the difference between one or more features of a feature group (e.g, skill feature group) of the candidate and one or more features of a feature group (e.g, skill feature group) of the job. Additionally, or alternatively, the model may determine a relationship as the difference between the feature group (e.g., as a whole) of the candidate and the feature group (e.g. as a whole) of the job. The model may output the fit and / or score indicating the difference. For example, the output may indicate a fit and / or a score of a difference between one or more feature groups and / or features of a candidate and one or more feature groups and / or features of ajob.

[0112]

[0095] A model may use weights, for example different weights, for one or more feature groups and / or features. For example, a model may determine that a feature group and / or feature should be weighted more than another feature group and / or feature. The model may determine that the feature group and / or feature should be weighted more than the other feature group and / or feature based on a user selection of weight(s) and / or training. For example, a model may be trained to determine that a feature group and / or feature should be weighted more than another feature group and / or feature based on labels (e.g, of training data) and / or synthetic training data.

[0113]

[0096] Input data, for example candidate and / or job input data, may include tokens. For example, tokens may include sentences, phrases, words, and / or characters of input data. Embeddings may include a vector representation of one or more tokens. A transformer neural network may embed a token into a vector, for example during training. Additionally, or alternatively, the transformer neural network may embed a token into a vector by converting the token into an embedding using a weight (e.g, weight matrix). The transformer neural network may additionally, or alternatively, embed positional information to a token to create an embedding. For example, the transformer neural network may add a position vector to a token. The transformer network may determine a position vector to add to a token based on the token position. For example, the transformer network may assign a position vector to an embedding of a token for the word scientist in the term (e.g. , title) data scientist. The position vector of the embedding for the word scientist and / or a position vector of an embedding for the word data, may avoid loss of positional meaning in an input term. For example, the position vector may include an indication of a relative position of a word in a sequence (e.g., of words).

[0114]

[0097] A contrastive learning head may determine contrastive loss, for example a contrastive loss function, based on embeddings. For example, the contrastive learning head may determine contrastive loss based on a similarity7(e.g., distance) between job and candidate embeddings. While a contrastive learning head is used as an example, other heads, for example for other losses as herein, may be used in a model.

[0115]

[0098] A neural network may receive tokens, data, information, features, and / or feature groups, for example by reading input data. For example, the input data may include a candidate resume, a job posting, and / or a webpage. The neural network may receive the input data at an input layer of the neural network. The input layer may include one or more layers before an encoder. The input layer may receive the input data by reading input data, for example from a candidate resume.

[0116]

[0099] FIG. 4A illustrates mean (e.g., average) pooling that may be used in a transformer architecture. The output embeddings of (e.g., all of) the tokens may be given equal importance and / or a mean value may be determined (e.g., calculated). The output may include the mean value. However, feature learning capability for the model may be restricted.

[0117]

[0100] FIGs. 4A and 4B illustrate examples of transformer architectures 400, 420 with example attention pooling. In example transformer architecture 400, one or more feature embeddings 402, 404, 406. 408, and / or 410 may be input. The one or more feature embeddings 402, 404, 406, 408, and / or 410 may be combined (e.g, added), for example to obtain an element wise summation 412. The transformer architecture 400 (e.g., model) may determine an average of the embeddings (e.g., feature embeddings 402, 404, 406, 408, and / or 410), for example based on the element wise summation 412. For example, the transformer architecture 400 (e.g., model) may determine the averaged embeddings 414 by dividing the element wise summation 412 by a number of embeddings (e.g., feature embeddings 402, 404, 406, 408, and / or 410).

[0118]

[0101] In an example transformer architecture 420, one or more feature embeddings 422, 424, 426, 428. and / or 430 may be input, for example into a weights network 434. The weights network 434 may include a weight network associated with each feature embedding input (e.g., 422, 424, 426, 428, and / or 430). Output of the weights network 434 may include a raw weight 436, for example for each embedding. For example, each embedding input (e.g., 422, 424, 426, 428, and / or 430) and / or each weight network 434 associated with each feature embedding input may have an associated raw weight 436.

[0119]

[0102] One or more raw weights 436 may (e.g., then) be sent to a softmax function 437, for example to set the raw weights 436 summation equal to 1. The softmax function 437 may output one or more weights 438 (e.g., one weight for each input feature embedding, associated weight network 434, and / or raw7weight 436). The one or more weights 438 output by the softmax function 437 may add to 1. Each of the one or more weights 438 output by the softmax function

[0120] 437 may then be associated with a feature embedding (e.g. ,440. 442, 444, 446. and / or 448). The weights 438 (e.g.. output by the softmax function 437) may signify the contribution of each feature group in the final embedding of the job and / or candidate. A softmax function may normalize weights, for example in a range from 0 to 1. The softmax function may indicate a probability of choosing a certain token (e.g.. in the range). Although a softmax function is used as an example, any suitable function may be used.

[0121]

[0103] Feature embeddings 440, 442, 444, 446, and / or 448 may correspond to (e g., be the same as) feature embeddings 422, 424, 426, 428, and / or 430 respectively. The transformer architecture 420 may determine an output embedding 450. For example, the transformer architecture 420 may multiply each feature embeddings 440, 442, 444, 446, and / or 448 with a respective weight

[0122] 438 (e.g., output by the softmax function 437).

[0123]

[0104] The transformer architecture 420 may utilize matrix multiplication (e.g., on the weights 438 and / or feature embeddings to determine the output embedding 450), for example across different words (e.g., in a job posting and / or a candidate resume) to identify which words are important. For example, for a feature (e.g., group), there may be N parameters. The attention mechanisms may identify which feature (e.g., group) contributes most to the selection of the candidate, for example in a ranked list of applicants. The attention mechanisms may increase an interpretability of the model which may improve explanation of the model, for example to a customer.

[0124]

[0105] Additionally, or alternatively, features may include features associated with ajob and features associated with ajob application. For example, a feature may include alternative job titles. A target job title may be senior software engineer and the candidate job application may include a title of software developer (e.g., from the candidate resume). A transformer architecture 400, 420 may generate an alternative job title, for example computer programmer.

[0125]

[0106] Other features (e.g., as herein) may include associated alternatives. A skill feature (e.g., associated with job information) may include Python, for example a job title of data scientist. The feature metadata may include data cleaning and / or data transformation. A skill feature (e.g. , associated with job information) may include Python, for example a job title of system administrator. The feature metadata may include system maintenance and / or scripting back-ups. Additionally, or alternatively, features, for example for the job information and / or the candidate information, may include metadata including occupation trends, skill trends, compensation (e.g.. typical compensation), geographic context, education, certification, and / or historical activity e.g., feedback). Historical activity’ may include an indication of prior activity from the job post, a similar job post, a job application, recruiter activity7, and / or hiring manager activity. A model may receive metadata, for example from job data and / or candidate data. The model may use the job data, the candidate data, and / or the metadata for training and / or for determining a candidate score. Metadata may be extracted, for example by a model from a second system. The model may extract the metadata from an ontology7, a database, and / or knowledge graph for example. The second system may be connected to the transformer architecture (e.g, model), for example by an approximate match (e.g, a seismic match of position information, title information, and / or skill information.

[0126]

[0107] The transformer architecture 420 (e.g, encoder) may take sentences which have inherent structure as input. However, data may be tabular, for example with features (e.g. columns) with respective values. Data may be converted into a format similar to that of a sentence, for example to gain structure. Tabular data may be converted into a sequential text data, for example with the feature first and followed by values. An example of converted data is show below7in Table 1.

[0127] Table 1

[0128]

[0108] O*NET is an occupational information network and is a comprehensive database created by the U.S. Department of Labor. O*NET provides detailed and up-to-date information on skills, knowledge, abilities, related job titles, work activities, and other key attributes required for various occupations in the U.S. workforce. Table 1 includes features including roles, skills, O*NET similar titles, domain, and ty pe. An example role may include machine learning engineer. Skills may include one or more of machine learning, deep learning, image processing, NLP, reinforcement learning, Bayesian graphical models, large language models. Python, and / or Pytorch. An example O*NET similar title may be data scientist. Domain may include computer and mathematical occupations. Finally, the type may include candidate. Each feature may be separated, for example by a separator token (SEP) such that each feature and associated value is separated from other features and associated values.

[0129]

[0109] Encoders may be configured to accept variable length input, for example with a constant context length. For example an encoder may accept a maximum of 256 or 512 tokens (e.g. depending on the model). A context length may be selected for a (e.g, each) specific entity. The context lengths may be combined to form an overall fixed length context input. These selections (e.g., data processing) may be for both candidate data and for the job data.

[0130]

[0110] For example for skills, the context length may be set to 50. The top 50 skills may be selected, for example based on the skill importance which may be calculated in context with the (e.g, entire) job or candidate details. For a sample (e.g, job or candidate), skill importance maybe sorted in descending order. The top 50 skills may be selected, for example if a sample has greater than 50 skills. If a sample has fewer skills than 50 for example, tokens may be appended until a context length of 50 skills is achieved. Example selections are depicted below.

[0131] Sample Skills: skill 1, skill 2, skill 3, skill 4, skill 5, . , skill n-1, skill n

[0132] Sample Skills: skill 1, skill 2, skill 3, skill 4, skill 5, . , skill 49, skill 50

[0133] Sample Skills: skill 1, skill 2, skill 3. skill 4. skill 5. skill x, . , [PAD],

[0134] [PAD]

[0135]

[0111] For example for titles, the context length may be set to 5 (e.g, the most recent 5 jobs of the candidate may be considered). The jobs may be sorted in increasing order of recency, for example with the most recent being last and the least recent being first. If the candidate has more than 5 jobs for example, the most recent 5 jobs may be selected and the rest omitted. If the candidate has fewer than 5 jobs for example, a context length of 5 may be maintained by adding padding tokens (e.g., at the beginning). The order may be maintained as such in order to utilize positional embeddings as disclosed herein. For the job input data for example, there may only be one title. The title may be added at the last position of the context length and / or prepended with padding tokens. Example selections are depicted below.

[0136] Sample Past Titles: title 1, title 2, title 3. title n-1. title n

[0137] Sample Past Titles: title n-4. title n-3, title n-2, , title n

[0138] Sample Past Titles: [PAD], [PAD], title 1. title 2, title 3

[0139]

[0112] Occupational data may be tagged with codes. Although O*NET is described as an example, any occupational data may be used with the models described herein and / or tagged as described herein. For example. ESCO data may be used.

[0140]

[0113] For O*NET similar titles for example, a job and / or a candidate may be tagged with a number (e.g., two) of the most probable O*NET codes. The titles of the O*NET codes may be input to the model. The O*NET code feature may be appended with one or more padding token, for example to reach a context length of two if less than two O*NET codes are tagged. For the domain feature, a job and / or a candidate may be tagged with a number (e.g.. two) of the most probable domains. The names of the domains may be given as input. The domain feature may be appended with one or more padding token, for example to reach a context length of two if less than two domains are tagged. Similarly, other features described herein may be processed to a context length (e.g., of two). An (e.g., sample) input may be made by appending (e.g., all) features following the respective feature name and separating the features, for example with separator tokens. Padding tokens may be processed with masks in the transformer model, for example such that such that the padding tokens do not affect the output prediction.

[0141]

[0114] The transformer may embed data positionally. Transformers may be positionally invariant, such that when a sentence is input to a transformer (e.g., and the order of w ords in the sentence is important), the transformer may have positional information encoded (e.g, explicitly) into the input data. Example techniques for injecting positional information into the transformer may include using sine / cosine waveforms with varying periodicity.

[0142]

[0115] Features as disclosed herein may not need any order information to be captured. For example, for skills with a context length of 50, the skills may be of any order, such that it is not important that skill X should come after skill Y. However, for some features, order may be beneficial. For example, order may be beneficial for a title feature (e.g., since a title may have an ordered meaning). The model may determine an order of recent titles, for example with the most recent title being in the last position. Positional information may be used (e.g., for input), for example when there is a change in a feature. For example, positional information may be input such that the model processes the next token(s) as next feature value(s). Positional embeddings may be described in Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L ., & Polosukhin, I. (2023a, August 2). Attention is all you need. arXiv.org. https: / / arxiv.org / abs / 1706.03762, which is hereby incorporated by reference. Positional embeddings may be generated using one or more of the example formulae:

[0143] An example Psuedocode to generate positional embeddings may include: Generate positional embeddings using above formulae Set the pointer to the 1st positional embedding Iterate through all the features

[0144] If feature is other than ‘title ’

[0145] Copy the current pointing positional embedding value to all the feature values

[0146] If the feature is ‘title ’

[0147] Iterate through each title copy the pointing positional embedding value to that title value Increment the pointer by 1 till last-1 title value

[0148] Increment the pointer by 1

[0149]

[0116] The training process may be iterated to update the parameters and / or hyperparameters 117 until an end condition is achieved. The end condition may be achieved when the output of the NN 109a is within a predefined threshold of the target output 115a.

[0150]

[0117] After the training process is complete, the trained NN 109a, or portions thereof, may be stored for being implemented by one or more devices. The trained NN 109a, or portions thereof, may be implemented in other downstream algorithms or processes, as may be further described herein. The trained NN 109a, or portions thereof, may be implemented on the same device on which the training was performed. The trained NN 109a, or portions thereof, may be transmitted or otherwise provided to another device for being implemented. For example, the NN 109b, 109c may include one or more portions of the trained NN 109a. The NN 109b and NN 109c receive respective input data 107b. 107c and to generate respective outputs 115b. 115c. The output 115b, 115c may be generated in one or more formats, such as a tensor, vector, a text format (e.g, a word, sentence, or other sequence of text), a numerical format (e.g, a prediction), an audio format, an image format, another data sequence format, and / or another output format. For example, the output 115b, 115c may be a prediction value. The output 115b, 115c may be aggregated at one or more devices for being further processed and / or implemented in other downstream algorithms or processes, as may be further described herein.

[0151]

[0118] Alternatively, or additionally, after the training process is complete, the trained parameters and / or tuned hyperparameters 117. or portions thereof, may be stored for being implemented by one or more devices. The trained parameters and / or tuned hyperparameters

[0152] 117, or portions thereof, may be implemented in other downstream algorithms or processes, as may be further described herein. The trained parameters and / or tuned hyperparameters 117, or portions thereof, may be implemented on the same device on which the training was performed. The trained parameters and / or tuned hyperparameters 117, or portions thereof, may be transmitted or otherwise provided to another device for being implemented. For example, transmitted or otherwise provided to another device or devices that may implement the NN 109b, 109c based on the trained parameters and / or tuned hyperparameters 117. For example, the NN 109b, 109c may be constructed at another device based on the trained parameters and / or tuned hyperparameters 117, or portions thereof. The NN 109b and NN 109c may be configured from the parameters / hyperparameters 117, or portions thereof, to receive respective input data 107b, 107c and to generate respective outputs 115b, 115c. The output 115b, 115c may be generated in one or more formats, such as a tensor, vector, a text format (e.g. , a word, sentence, or other sequence of text), a numerical format (e.g., a prediction), an audio format, an image format, another data sequence format, and / or another output format.

[0153]

[0119] The output 115b. 115c may be aggregated at one or more devices for being further processed and / or implemented in other downstream algorithms or processes, as may be further described herein. The input data 107b, 107c may include job input data. For example the job input data 107b, 107c may include one or more of (e.g., required) skills, job title, experience level, and / or location. Additionally, or alternatively, the input data 107b, 107c may include candidate input data, which may be received from a candidate resume. For example, candidate input data 107b, 107c may include one or more of e.g., past) job titles, skills, experience, and / or location. The neural network 109b, 109c may process and leam from the relationships between job (e.g, input) data and candidate (e.g, input) data. The output 115b, 115c may include a prediction of a candidate for a (e.g, given) job. For example, the output 115 may include a suitability score and / or classification (e.g., of the candidate for the job). The model may evaluate a candidate by computing a candidate embeddings and comparing the candidate embeddings to the job requirements. A candidate with candidate embeddings closer to the job embeddings may have a higher suitability score and / or classification.

[0120] The model may utilize input data 107b, 107c that grounds the model with respect to the labor market, for example by including standardized occupation input data 107b, 107c to the neural network 109a, 109c. The standardized occupation input data 107b, 107c may be from O*NET, for example when a job (e.g., posting) does not provide much job data. The standardized occupation input data 107b, 107c may be preprocessed prior to being input to the neural network 109a, 109c.

[0154]

[0121] O*NET structures occupational information into multiple hierarchical levels, for example that address different information. Example information include one or more of job family, category, and / or occupation code. Example models disclosed herein may be trained as one or more of a job family for jobs, occupation code for jobs, occupation (e.g., job) family for candidates, and / or occupation code for candidates.

[0155]

[0122] A transformer architecture (e.g., model) may be used for multi-objective training. For example, multi-objective training may create multiple outcomes for use. Example outcomes may include accurate classification and embeddings. The transformer architecture (e.g., model) may use contrastive learning to determine the outcomes.

[0156]

[0123] Multi-objective training in a neural network may include optimizing a model for multiple objectives simultaneously. The objectives may be unrelated and / or competing in some examples. A transformer architecture (e.g., model) as herein may perform multi-objective training in a human resources context, for example for candidate scoring. A neural network may be trained to minimize a (e.g., one) loss function. However, a neural network herein may be trained with a plurality of loss functions. For example, each loss function may correspond to a (e.g., different) outcome and / or performance metric. A neural network may combine a plurality of loss functions into one function, for example a scalarized objective function. The function may include a candidate score (e.g., a score associated with a suitability of the candidate for the job).

[0157]

[0124] A model trained for job family for jobs may predict the probability of each job family for a job. for example given input including one or more of a free text title, a free text description, and / or attributes of the job. A model trained for occupation code for jobs may predict the probability of an occupation code for the j ob, for example given input including one or more of a free text title, a free text description, and / or attributes of the job. A model trained for occupation (e.g., job) family for candidates may predict the probability of each job family for the candidate, for example given input including one or more of a free text of job title history, a free text of the candidate resume, and / or attributes of the candidate profile. A model trained to occupation code for candidates may predict the probability of an occupation code for a candidate, for example given input including one or more of a free text of job title history, a free text of the candidate resume, and / or attributes of the candidate profile.

[0158]

[0125] FIG. 5 illustrates an example of a model architecture 500. The model architecture 500 may be used for a delayed interaction, for example delay of input components interaction until a later layer. By delaying interaction of input components until a later layer, model accuracy may be increased, computation speed may be increased, and / or computing resources needed may be decreased. For example, early interaction between data, for example embeddings, may not significantly affect model accuracy. Early interaction between certain data, for example, may not improve model accuracy at all. Interaction of candidate data (e.g., embeddings) and job data (e.g., embeddings) may not improve model accuracy at early layers. For example, candidate data (e.g., embeddings) and job data (e.g., embeddings) may not improve model accuracy before cross attention of candidate data (e.g., embeddings) and job data (e.g., embeddings) may not improve model accuracy. Additionally, or alternatively, model accuracy may not be improved by early interaction between certain feature groups (e.g., embeddings) and / or features (e.g, embeddings). A model may select which data (e.g., embeddings), for example groups (e.g, embeddings) and / or features (e.g, embeddings) should interact at certain layers. For example, a model may determine that candidate feature groups (e.g., embeddings) and job feature groups (e.g, embeddings) may not interact before cross attention (e.g., layer(s)). Delaying interaction may result in a more accurate model, a faster computation, and / or avoidance of unnecessary use of computing resources.

[0159]

[0126] The model architecture 500 may be used in a human resources context, for example for candidate scoring. In the example model architecture 500 job data 502 may be input into the model architecture 500. For example, the job data 502 may be input into an encoder layer of the model architecture 500. The model architecture 500 may include a self attention block 504 (e.g., attention layer). The job data 502 may be input to the self attention block 504. The self attention block 504 may weigh the importance of tokens of the job data 502.

[0160]

[0127] The self attention block 504 may output one or more embeddings 506, for example based on the weight of the importance of (e.g, each of the) tokens of the job data 502. The one or more embeddings 506 may include a skills embedding, a titles embedding, an experience embedding, an experience embedding, a certification embedding, and / or a licenses embedding.

[0161]

[0128] Candiate data 510 may additionally, or alternatively, be input to the model architecture 500, for example at the encoder layer. The model architecture 500 may include a (e.g., another) self attention block 512. The candidate data 510 may be input to the self attention block 512. The self attention block 512 may weigh the importance of tokens of the candidate data 510.

[0162]

[0129] The self attention block 512 may output one or more embeddings 514, for example based on the weight of the importance of (e.g.. each of the) tokens of the candidate data 510. The one or more embeddings 514 may include a skills embedding, a titles embedding, an experience embedding, an experience embedding, a certification embedding, and / or a licenses embedding.

[0163]

[0130] Additionally, or alternatively, a model may generate an embedding. For example, a model may generate an embedding by processing input data (e g., candidate data / tokens and / or job data / tokens) to

[0164]

[0131] The one or more embeddings 506 may be input such that the one or more embeddings 506 of the job data 502 interact with the one or more embeddings 514 of the candidate data 510. For example, the interaction of the one or more embeddings 506 of the job data 502 and the one or more embeddings 514 of the candidate data 510 may be at a later layer, for example resulting in increased model accuracy, increased computation speed, and / or decreased needed computing resources. The one or more embeddings 506 of the job data 502 may interact with the one or more embeddings 514 of the candidate data 510 such that one or more cross attention 508 is determined. For example, the one or more cross attention 508 may include a skills cross attention, a titles cross attention, an experience cross attention, a certification cross attention, and / or a licenses cross attention. The attention of the one or more embeddings 506 job data 502 may be higher than the attention of the one or more embeddings 514 of the candidate data 510 to determine the one or more cross attention 508.

[0165]

[0132] The one or more embeddings 514 may be input such that the one or more embeddings 514 of the candidate data 510 interact with the one or more embeddings 506 of the job data 502. For example, the interaction of the one or more embeddings 514 of the candidate data 510 and the one or more embeddings 506 of the job data 502 may be at a later layer, for example resulting in increased model accuracy, increased computation speed, and / or decreased needed computing resources. The one or more embeddings 514 of the candidate data 510 may interact with the one or more embeddings 506 of the job data 502 such that one or more cross attention 516 is determined. For example, the one or more cross attention 516 may include a skills cross attention, a titles cross attention, an experience cross attention, a certification cross attention, and / or a licenses cross attention. The attention of the one or more embeddings 514 of the candidate data 510 may be higher than the attention of the one or more embeddings 506 of the job data 502 to determine the one or more cross attention 508.

[0133] The model architecture 500 may input the one or more cross attention 508 and / or the one or more cross attention 516, for example to an entity level Maximum Similarity Function MaxSim operation 518. The entity level MaxSim operation 518 may output values 520 associated with one or more of the embeddings 506 and / or the embeddings 514. For example the values 520 may include one or more of a skills MaxSim value, a title MaxSim value, an experience MaxSim value, an education MaxSim value, and / or a certification MaxSim value. While MaxSim is used as an example herein, any operation may be used in a model.

[0166]

[0134] A MaxSim function for Contextualized Late Interaction over BERT (ColBERT) may use a late interaction mechanism where embeddings for query and document tokens are generated independently. The MaxSim function may (e.g., then) calculate a similarity' between the embeddings for query and document tokens, for example during a ranking phase. The MaxSim function may find the maximum similarity between any query token embedding and any document token embedding.

[0167]

[0135] The MaxSim function may generate token embeddings, for example query' embeddings and / or document embeddings. For a query embedding for example, for a given query $Q$. each token $q_i$ may be mapped to a dense vector embedding $e_{q_i}$, for example using a pretrained BERT model. For a document embedding for example, for a given document $D$, each token $dj$ may be mapped to a dense vector embedding $e_{dj}$, for example using the same BERT model.

[0168]

[0136] The MaxSim function may use late interaction, for example by instead of computing a single similarity' score between the entire query' and document, computing a similarity score for each pair of query token embedding and document token embedding. The MaxSim function may determine an overall similarity score between the query $Q$ and the document $D$, for example by taking the sum of the maximum similarities for each quety token. Specifically, for each query' token $q_i$, a contribution to the overall score may be the maximum similarity achieved with any token in the document $D$.

[0169]

[0137] An example MaxSim function may be expressed as:

[0170] $Sim(Q, D) = \sum_{i=l}A{|Q|] \max_{j=l}A{|D|] (e_{q_i} \cdot e_{dj})$, where:

[0171] $|Q|$ may be the number of tokens in the query,

[0172] $|D|$ may be the number of tokens in the document, $e_{q_i } $ may be the embedding for the $i$-th query token,

[0173] $e_{d_j }$ may be the embedding for the $j $-th document token, and / or

[0174] $\cdot$ may represent the dot product (e.g, cosine similarity may be used, which may be a normalized dot product).

[0175]

[0138] A MaxSim function may achieve granular matching, robustness to noise, explainability, and / or efficiency. Granular matching may allow for fine-grained matching between specific query terms and relevant document terms, for example even if they are far apart in the document. Robustness to noise may include more robustness to irrelevant terms in either the query or the document, as for example the focus may be on the strongest matches. Explainability may include the contribution of each query term to the overall score being observed, for example offering interpretability. Efficiency may include the parallel nature of modem hardware and optimizations (e.g., vector quantization), for example despite multiple dot products. A MaxSim function (e.g., ColBERT MaxSim function) may capture a highly detailed understanding of semantic relevance, for example by allowing each query term to hunt for a best match within the document, and / or (e.g. then) aggregating these best matches.

[0176]

[0139] The values 520 may be input to a classification head 522. The classification head 522 may determine a fit, for example of the candidate to the job. The classification head 522 may determine a fit of the candidate to the job based on one or more of the values 520. For example, the classification head 522 may determine a similarity between the candidate (e.g, candidate data 510) and the job (e.g., job data 502). The classification head 522 may determine the similarity between the candidate and the job based on the one or more values 520. The model architecture 500 may output the fit prediction at 524. The fit prediction 524 may indicate a prediction of a fit of the candidate to the job, for example numerically.

[0177]

[0140] The classification head 522 may compare the observed value of a target variable with a prediction generated by the rest of the network to compute a loss function. For multi-objective training for example, a model may include a plurality of heads 522. Each head 522 may be for a different purpose. For example, there may be a classification head (e.g., 310 and / or 324), an embedding head, and / or a contrastive learning head (e.g., 312).

[0178]

[0141] A classification head 522 may use binary cross-entropy loss. Binary cross-entropy loss may quantify a difference between a label and a predicted probability of the model. However, in some examples the classification head 522 may use hinge loss and / or focal loss. Hinge loss may be used when the predictive model does not use (e.g., need) probability calibration. Hinge loss may include a function maximizes a margin between data classes, for example by penalizing incorrect and / or inaccurate predictions. Focal loss may be used when there is imbalanced data for example. Focal loss may be similar to binary' cross-entropy loss. Focal loss may include a modification to cross-entropy loss, for example that decreases the weight of examples that are more easily classified.

[0179]

[0142] A head may use circle loss. For example, a contrastive learning head (e.g., 312) may use circle loss. Circle loss may include maximization of in class similarities and minimization of between class similarities, for example by weighing similarity scores based on optimization. A model may utilize circle loss (e.g., in a contrastive learning head) to manage unequal distribution of positive and negative examples during training. Triplet-based loss functions may be used in a model, for example to optimize a batch of observations simultaneously. For example, tripletbased functions may include noise contrastive estimation, InfoNCE, NT-Xent, and / or SupCon.

[0180]

[0143] A transformer architecture may combine a cross-entropy loss with contrastive loss, for example using scalarization. For example, the transformer architecture may use cross-entropy loss to train a model that is an accurate predictor of which candidate fits with a job. The transformer architecture may use contrastive loss to train a model that learns an embedding representation ofjob(s) (e.g., job(s)) and candidate(s).

[0181]

[0144] Delayed interaction in a neural network, for example model architecture 500, may include a delay in different input components (e.g, query tokens and document tokens) being deferred until a later stage of the model (e.g., 508 and / or 516). Delayed interaction models may process each input (e.g., 502 and 510) independently to generate dense vector representations for their respective components (e.g., 506 and 514). These representations (e.g., 506 and 514) may then be compared and / or interacted together (e.g.. 508 and / or 516), for example to produce a similarity score or prediction (e.g., 524). This approach can lead to faster training and more accurate predictions by reducing the computational overhead of early, dense interactions. Therefore, model accuracy may be increased, computation speed may be increased, and / or computing resources needed for training may be decreased.

[0182]

[0145] FIG. 6 illustrates example input 602 to an example model 600. For example, inputs 602 to a (e.g., predictive model) may include one or more of job title 604, skill 606 (e.g. , group), job description 608, other job attributes, and / or other candidate attributes. The (e.g, predictive) model 600 may predict the probability of a (e.g., each) possible job family and / or occupation code. For example, the model 600 may classify a job family 610. Additionally, or alternatively, the model 600 may classify' an occupation 620.

[0146] The model 600 may utilize job family data to classify (e.g, the candidate) to ajob family 610 and / or occupation 620. While any job family data may be utilized by the model 600, O*NET is used as an example. Job family data may be included in an occupational database, for example O*NET or ESCO. O*NET may contain 23 possible job families. For example job families may include a management family 612. a business and financial operations family 614, a computer and mathematical family 616, and / or a healthcare support family 618. The predictive model 600 may produce a probability estimate for one or more of the (e.g., 23 possible) job families. The model 600 may select a subset of the possible job families. For example, the subset may include the most probable job family or a number, K of the most probable job families. The model may classify one or more job (e.g, the subset) at the job family classifier 610.

[0183]

[0147] The model 600 may utilize occupational data to classify (e.g., the candidate) to ajob family 610 and / or occupation 620. While any occupational data may be utilized by the model 600, O*NET is used as an example. O*NET may contain over 1000 possible occupations. For example, occupations may include management occupations 622, business and financial occupations 624, computer and mathematical occupations 626, and / or healthcare and support occupations 628. The predictive model 600 may produce a probability estimate for one or more of the (e.g. possible) occupations. The model 600 may select a subset of the possible occupations. For example, the subset may include the most probable occupation or a number, K of the most probable occupations. The model may classify one or more occupations (e.g., the subset) at the occupation classifier 620.

[0184]

[0148] The model 600 may determine an output 630, for example based on the determination of one or more probable job families (e.g., at the job family classifier 610) and / or based on the determination of the one or more probable occupations (e.g., at the occupation classifier 620). For example, the model 600 may determine the output 630 as a most probable occupation 632. The model 600 may determine the most probable occupation 632 based on respective weights of the one or more probable job families (e.g., at the job family classifier 610) and / or the one or more probable occupations (e.g., at the occupation classifier 620). A weight may be determined by a user (e.g, subject matter expert). A weight may additionally, or alternatively, be determined by a model, for example to optimize performance metnc(s) on data (e.g. a dataset). An example dataset may include a validation dataset. Performance metrics may include metrics of machine learning, for example confusion matrix metrics, F-score, and / or loss. The determined one or more possible job families (e.g., at the job family classifier 610), the determined possible one or more occupations (e.g, at the occupation classifier 620), and / or the most probable occupation 632 may be incorporated as features in a model, for example a Fitformer model. Features may be incorporated in a model as categorical (e.g, indicating each job family or occupation as a distinct category) and / or as natural language strings (e.g, indicating each job family or occupation as an English or other human language descriptor that can be interpreted by a statistical language model), for example, the same language model that interprets skill names and / or job title names.

[0185]

[0149] FIG. 7 illustrates example transformers 700, 730. The example transformers 700, 730 may take sequential data as input such that the (e.g. sequential data) input may be embedded to obtain input embeddings. Additionally, or alternatively, inputs may be positionally embedded. In some examples input may be masked (e.g.. by mask information). The encoder of the model (e.g, transformer architecture) may include one or more of a multi-head self attention layer, a layer norm, a feedforward with ReLU activation, and / or a (e.g, second) layer norm. The output may be trained on a language modeling task.

[0186]

[0150] In example transformer 700, sequential data 702 may be input. The transformer 700 may determine input embeddings 704, for example based on the sequential data 702. Additionally, or alternatively, positional embeddings 706 may be input into the transformer 700. A mask 708 may mask the input data, for example the sequential data 702, the input embeddings 704, and / or the positional embeddings 706. The mask 708 may be specific to the sequential data 702. The mask 708 may control which information (e.g, data) is available to self-attention 712 and / or may reflect domain expertise about the structure of the data. For example, the mask 708 may represent that a job posting that requires one skill, does not necessarily require a second skill, and / or that a candidate whose employment history' contains one job title does not necessarily contain a second title.

[0187]

[0151] The transformer 700 may include an encoder 710. The encoder 710 may include one or more of a multi-head self attention layer 712, a layer norm 714, a feedforward with rectified linear unit (ReLU) activation 716, and / or a (e.g., second) layer norm 718. The encoder 710 and / or transformer 700 may output at 720. The output 720 may include a model trained on a language modeling task. The model (e.g, output) may be utilized for in a human resources context (e.g, as herein).

[0188]

[0152] In example transformer 730, tabular data 732 may be input. For example, tabular data 732 converted to sequential data may be input. Additionally, or alternatively, positional embeddings for tabular data 736 may be input into the transformer 730. A mask 738 may mask the input data, for example the tabular data (e.g., converted to sequential data) 732, the input embeddings 734. and / or the positional embeddings for tabular data 736. The mask 738 may be specific to the tabular data 732.

[0189]

[0153] The transformer 730 may include an encoder 740. The encoder 740 may include one or more of a multi-head self attention layer 742, a layer norm 744. a feedforward with Leaky ReLU activation 746, and / or a (e.g., second) layer norm 748. The encoder 740 and / or transformer 740 may output at 730. The output 730 may include a model trained on a contrastive task. The model (e.g. , output) may be utilized for in a human resources context (e.g. , as herein).

[0190]

[0154] Additionally, or alternatively, padding tokens may be placed such that tabular data may be utilized. Positional embeddings may be adjusted such that the embedding indicates a change in a feature group input and / or a role transition (e.g.. of work experience) of a candidate. Feature representations using attention pooling may be combined, for example at the last layer (e.g, of Fitformer). Tabular data, for example converted to sequential data may be input such that the (e.g., tabular data) input may be embedded to obtain input embeddings. Additionally, or alternatively, inputs (e.g., sequential data 702, input embeddings 704, tabular data converted to sequential data 732, and / or input embeddings 734) may be positionally embedded. In some examples input may be masked (e.g, by mask information for tabular data).

[0191]

[0155] The AI / ML models and / or algorithms described herein may be implemented on one or more devices. Example networks in which AI / ML may be distributed may include federated networks. A federated network may include a decentralized group of devices that each include AI / ML. As shown in FIG. 1C, the AI / ML 109b and AI / ML 109c may be distributed across separate devices. Though FIG. 1C shows two models (e.g., AI / ML 109b and AI / ML 109c), any number of models may be implemented across any number of devices. The AI / ML may be implemented for collaborative learning in which the AI / ML is trained across multiple devices. In another example, the AI / ML may be trained at a centralized location or device and one or more portions of the AI / ML, or trained parameters and / or tuned hyperparameters, may be distributed to decentralized locations. For example, updated parameters or hyperparameters may be sent to one or more devices for updating and / or implementing the AI / ML thereon. The AI / ML may be deployed in a production environment, for example where the AI / ML model is integrated in a recruitment system. The AI / ML model may be configured to process (e.g., new) jobs and candidates (e.g, resumes), for example in real-time.

Claims

CLAIMSWhat is Claimed:

1. At least one computer-readable medium having stored thereon instructions that, when executed by at least one processor, are configured to cause the at least one processor to: receive a plurality of feature groups in a transformer neural network, the plurality of feature groups comprising a plurality of features; determine a first relationship between each of the feature groups at one or more first layers of the transformer neural network using respective first weights; determine a second relationship between each of the plurality of features at one or more second layers of the transformer neural network using respective second weights; and output an indication of the first relationship between the plurality of feature groups or the second relationship between the plurality of features.

2. The at least one computer-readable medium of claim 1, wherein the instructions, when executed, are further configured to cause the at least one processor to: generate an embedding of each feature group of the plurality of feature groups with respective positional information; and generate an embedding of each feature of the plurality of features with respective positional information.

3. The at least one computer-readable medium of claim 1, wherein the instructions, when executed, are further configured to cause the at least one processor to: determine the first weights using matrix multiplication across each embedded feature group; and determine the second weights using matrix multiplication across each embedded feature.

4. The at least one computer-readable medium of claim 1, w herein the instructions, when executed, are further configured to cause the at least one processor to: determine the first weights or the second w eights using a triplet loss function.

5. The at least one computer-readable medium of claim 1, wherein the instructions, when executed, are further configured to cause the at least one processor to:determine the first weights or the second weights using a supervised contrastive loss function.

6. The at least one computer-readable medium of claim 1, wherein the instructions, when executed, are further configured to cause the at least one processor to: determine the first weights or the second weights using a circle loss function.

7. The at least one computer-readable medium of claim 1, wherein the plurality of feature groups comprise feature groups associated with a candidate for a position, and the features comprise features associated with the candidate for the position, and wherein the indication comprises a score associated with a suitability of the candidate for the position.

8. The at least one computer-readable medium of claim 1, wherein the plurality of feature groups comprises: a plurality of feature groups associated with a plurality of jobs; a first plurality of feature groups associated with a first plurality of candidates for one or more of the plurality of jobs; and a second plurality of feature groups associated with a second plurality of candidates for one or more of the plurality of jobs.

9. At least one computer-readable medium having stored thereon instructions that, when executed by at least one processor, are configured to cause the at least one processor to: receive a plurality of feature groups in a transformer neural network, wherein the plurality of feature groups are associated with standardized job feature groups, wherein the standardized job feature groups are associated with an occupational database; receive a plurality of features in the transformer neural network, wherein the plurality' of features are associated with standardized job features, wherein the standardized job features are associated with the occupational database; process each of the plurality of feature groups at one or more first layers of the transformer neural network to determine a relationship between each of the plurality of feature groups; process each of the plurality of features at one or more second layer of the transformer neural network to determine a relationship between each of the plurality of features; andoutput an indication of the relationship between at least two of the plurality of feature groups or the relationship between at least two of the plurality of features.

10. The at least one computer-readable medium of claim 9, wherein the indication comprises an indication of the relationship between a predetermined number of feature groups or the relationship between a predetermined number of features.

11. The at least one computer-readable medium of claim 9, wherein the standardized job feature groups or the standardized job features comprise standardized feature groups or features from occupational data.

12. A method of training a model with a transformer neural network, the method comprising: selecting data associated with a candidate for a job; selecting job data associated with the job; embedding the data associated with the candidate for the job with a label; inputting the data associated with the candidate for the job and the data associated with the job into a neural network; determining a classification loss based on the data associated with the candidate for the job and the data associated with the job; determining a contrastive loss based on the data associated with the candidate for the job and the data associated with the job; and determining a multi-objective loss based on the classification loss and the contrastive loss.

13. The method of claim 12, wherein the multi-objective loss comprises an indication of a suitability of the candidate for the job.

14. The method of claim 12, wherein the label consists of a positive label.

15. A method of training a model with a transformer neural network, the method comprising: randomly selecting a feature from a plurality of features, wherein the plurality of features comprise features associated with a candidate for a job, and wherein the plurality of features comprises a plurality of features with a positive label;randomly changing data from the selected feature of the plurality of features to generate synthetic data, the generation of the synthetic data based on the feature including the positive label, the synthetic data being less positive than the feature including the positive label; inputting the synthetic data as training data into the transformer neural network; and training the transformer neural network using the synthetic data.

16. The method of claim 15, wherein the plurality of features comprises: a plurality of features associated with a plurality of jobs; a first plurality of features associated with a first plurality of candidates for one or more of the plurality of jobs; and a second plurality7of features associated with a second plurality of candidates for one or more of the plurality7of jobs.

17. The method of claim 15. wherein the plurality of features comprises a first plurality of features and the synthetic data comprises first synthetic data, and wherein the method comprises: randomly selecting a feature from a second plurality of features including a negative label, wherein the second plurality of features comprise features associated with a candidate for a job, and wherein the second plurality of features comprises a plurality of features with a negative label; randomly changing data from the selected feature of the plurality of features to generate second synthetic data, the generation of the second synthetic data based on the feature including the negative label, the second synthetic data being more than the feature including the negative label; inputting the second synthetic data as training data into the transformer neural network; and training the transformer neural network using the second synthetic data.

18. A method of training a model with a transformer neural network, the method comprising: selecting data associated with a candidate for a job; selecting data associated with the job; inputting the data associated with the candidate for the job and the data associated with the job into a neural network; receiving metadata from at least one of the data associated with the candidate for the job or the data associated with the job; andinputting the data associated with the candidate for the job, the data associated with the job, and the metadata from at least one of the data associated with the candidate for the job or the data associated with the job into a neural network into the transformer neural network.

19. At least one computer-readable medium having stored thereon instructions that, when executed by at least one processor, are configured to cause the at least one processor to: receive data associated with a job; receive data associated with a candidate for the job; determine at least one embedding for at least one feature of the data associated with the job, wherein the determination of the at least one embedding for the at least one feature of the data associated with the job is based on the data associated with the job; determine at least one embedding for at least one feature of the data associated with the candidate for the job. wherein the determination of the at least one embedding for the at least one feature of the data associated with the candidate for the job is based on the data associated with the candidate for the job; determine, based on the at least one embedding for at least one feature of the data associated with the job and the at least one embedding for at least one feature of the data associated with the candidate for the job, at least one cross attention value associated with the at least one feature of the data associated with the candidate and at least one cross attention value associated with the job; and determine an indication of a suitability of the candidate for the job based on the at least one cross attention value associated with the at least one feature of the data associated with the candidate and the at least one cross attention value associated with the job.

20. The at least one computer-readable medium of claim 19, w herein the determination of the at least one embedding for the at least one feature of the data associated with the candidate for the job is not based on the data associated with the job, and wherein the determination of the at least one embedding for the at least one feature of the data associated with the job is not based on the data associated with the candidate for the job.

Images