Resume recommendation method based on large-scale pre-training language model and contrastive learning

Abstract

The application discloses a resume recommendation method based on a large-scale pre-training language model and contrast learning, and the resume recommendation method learns text representation on a large amount of unlabeled corpus by using a self-supervised learning mode through a pre-training language model, and can better understand language and obtain better text feature vectors compared with a traditional word vector model. By fully utilizing the advantage of the pre-training language model, the resume and the post description text are represented by using the pre-training language model respectively, the text feature vectors are obtained, and then the fine-tuning training based on the contrast learning is carried out, the distribution in the implicit feature space where the feature vectors are located is optimized, and then the text representation is enhanced, and the representation effect and matching accuracy of the traditional model can be far surpassed on the same or smaller labeled data set size.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer software, and particularly relates to a resume recommendation method based on a large-scale pre-training language model and contrastive learning. BACKGROUND

[0002] A resume recommendation system is applied in various scenarios. For example, for an enterprise, a resume matching a certain post can be quickly obtained from a large number of resumes. The core method of the current resume recommendation is to calculate the similarity between the resume and the post description, and the resume with a greater similarity to a certain post description is taken as the recommendation result for the post.

[0003] For the similarity calculation between the resume and the post description, the existing technology mainly adopts a keyword extraction-based technology, that is, a keyword set contained in the resume is first extracted, and the matching degree is calculated with a keyword set contained in the post description. Some recent works introduce a deep learning-based model to model the resume and the post, represent the resume text and the post description text in the form of a vector, and train the deep learning model to learn better feature vectors according to the matching degree of the manually labeled resume and post description, and then calculate the distance between the vectors as the similarity between the resume and the post.

[0004] For the related technology based on keyword extraction, the matching mechanism mainly depends on the keywords, rather than the text with linguistic structure. This leads to the loss of too much semantic information when this kind of technology matches the resume and the post, and makes it difficult to meet the fine-grained matching demand. Generally, the deep learning-based model highly depends on the size of the manually labeled data set, and it is difficult to play its advantage over the traditional statistical model for low-resource tasks. Moreover, the sample utilization efficiency of the general deep learning model is often not high, which leads to the fact that it cannot fully utilize the rich semantic connotations contained in the resume and post description text.

[0005] Therefore, the existing technology has defects and needs to be improved. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a resume recommendation method based on a large-scale pre-training language model and contrastive learning.

[0007] The technical scheme of the present application is as follows: a resume recommendation method based on a large-scale pre-training language model and contrastive learning, comprising: step S1: using a pre-training language model RoBERTa to obtain feature representation vectors of resume texts and job descriptions, corresponding to output resume vectors and job vectors respectively; based on a resume-job description corpus, pre-training the model in a self-supervised manner to make the model familiar with the corpus; step S2: using a weighted contrastive learning training on the pre-trained language model; step S3: inputting the resumes in the resume library into the language model to obtain a resume vector library, and inputting the job description to be recommended into the language model to obtain a job vector, taking the top k resume vectors in the feature space that are most adjacent to the job vector as the recommendation, and returning the result.

[0008] Applied to the above technical scheme, in the resume recommendation method based on a large-scale pre-training language model and contrastive learning, in step S1, when using the pre-training language model RoBERTa to obtain the feature representation vectors of the resume texts and the job descriptions, the input is pure text of the resume and the job description.

[0009] Applied to each of the above technical schemes, in the resume recommendation method based on a large-scale pre-training language model and contrastive learning, in step S1, when performing self-supervised pre-training, a masking language model is used to perform self-supervised pre-training on the language model's resume-job description corpus.

[0010] Applied to each of the above technical schemes, in the resume recommendation method based on a large-scale pre-training language model and contrastive learning, in step S1, in the self-supervised pre-training, the probability distribution predicted by the model is optimized based on cross-entropy loss.

[0011] Applied to each of the above technical schemes, in the resume recommendation method based on a large-scale pre-training language model and contrastive learning, in step S2, when performing weighted contrastive learning training, the positive sample pair is a resume-job description feature vector pair in different recommendation stages annotated by human in the headhunting company talent recommendation record, and the negative sample pair is other samples in the randomly combined training batch; and when performing contrastive learning training, the language model aggregates the positive sample pair in the implicit feature space and distances the negative sample pair.

[0012] Applied to each of the above technical schemes, in the resume recommendation method based on a large-scale pre-training language model and contrastive learning, in step S2, each resume-job description feature vector pair corresponds to a recommendation process state, and different recommendation process states give corresponding positive sample pairs different weights in contrastive learning loss calculation.

[0013] Applied to each of the technical solutions above, the resume recommendation method based on the large-scale pre-training language model and the contrast learning, in step S2, the contrast learning objective function adopted is: Wherein, B is the number of resume-post description feature vector pairs for one batch of contrast learning objective function, is the weighted contrast learning loss of the resume vector of any number i in B, is the weighted contrast learning loss of the post vector of any number i in B.

[0014] Applied to each of the technical solutions above, the resume recommendation method based on the large-scale pre-training language model and the contrast learning, in step S2, the cosine distance is used to calculate the distance between two feature vectors.

[0015] Applied to each of the technical solutions above, the resume recommendation method based on the large-scale pre-training language model and the contrast learning, in step S2, the weight implementation strategy of the recommendation process state of any two resume-post description feature vector pairs adopts the strategy of frequency-aware soft weight.

[0016] Applied to each of the technical solutions above, the resume recommendation method based on the large-scale pre-training language model and the contrast learning, in step S3, the cosine distance between the post vector and all vectors in the implicit feature space in the resume vector library is calculated by the language model, the first k resume vectors most adjacent to the post vector in the feature space are taken as the recommendation, and the result is returned.

[0017] The beneficial effects of the present application are:

[0018] The present application learns text representation on a large amount of unlabeled corpus by using self-supervised learning method through pre-training language model, and can better understand language and obtain better text feature vector compared with traditional word vector model. By fully utilizing the advantage of the pre-training language model, the resume and the post description text are represented by the pre-training language model respectively, and then the fine-tuning training based on the contrast learning is carried out after obtaining the text feature vector, the distribution in the implicit feature space of the feature vector is optimized, and then the text representation is enhanced, and the representation effect and matching accuracy far superior to traditional model can be obtained on the same or smaller labeled data set size.

[0019] The present application has the following advantages:

[0020] 1、The present application uses a pre-trained language model to perform text vector representation of resumes and job descriptions, which can effectively utilize an unlabeled resume-job description corpus relative to traditional methods based on word vectors or embeddings, fully utilize the text connotation in the resume-job description, and does not require additional branch operations, further simplifying the data processing method.

[0021] 2、The training method of the present application designed to give contrast learning can more efficiently utilize training data relative to the traditional text classification-based training method, can more significantly distinguish irrelevant resume and job text and aggregate mutually matching resumes and job text, and further improve the accuracy of resume recommendation. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1a An illustrative diagram for using a pre-trained language model RoBERTa to obtain feature representation vectors of text in the present application;

[0023] Figure 1b An illustrative diagram for using weighted contrast learning to train in the present application;

[0024] Figure 1c An illustrative diagram for using a trained language model to perform resume recommendation in the present application;

[0025] Figure 2 An illustrative diagram for using a masked language model to perform self-supervised pre-training on a resume-job description corpus in the present application. DETAILED DESCRIPTION

[0026] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0027] The present embodiment provides a resume recommendation method based on a large-scale pre-trained language model and contrast learning, which improves the text representation effect under limited data amount, and designs a training method based on contrast learning to further improve the text representation effect and training efficiency under limited data amount.

[0028] The framework of the resume recommendation method based on a large-scale pre-trained language model and contrast learning mainly includes three parts, which are as follows:

[0029] 1. Pre-trained language model and text vector representation, i.e., self-supervised pre-training of the language model;

[0030] We use a pre-trained language model RoBERTa to obtain feature representation vectors of text. As shown in FIG. 1, the RoBERTa model is used to obtain the feature representation vectors of the text. Figure 1aAs shown, the model accepts plain text input from resumes and job descriptions and outputs feature vector representations of the text. Because large-scale pre-trained language models are pre-trained on general corpora, some domain-related words that frequently appear in resume and job description texts are labeled as rare tags, which may affect the model's text understanding ability.

[0031] At this point, fine-tuning the language model on downstream task corpora can effectively improve the model's performance on downstream tasks. To improve the ability of large-scale pre-trained language models to understand and represent resume-job description texts, a masked language model is used for self-supervised pre-training on a resume-job description corpus.

[0032] Specifically, the masked language model task is similar to a cloze test. We represent the model's input as a sequence of labels X = {x1, x2, ..., x...}. n Randomly sample the index set I∈{1,2,...,n}, and include all elements belonging to the set {x} i The label of |i∈I} is replaced with the label [MASK], resulting in a new label sequence X′={x1,[MASK],...x i-1 [MASK], x i+1 , ..., x n The model receives X′ and represents it as an implicit state vector sequence H = {h1, h2, ..., h...}. i-1 h i h i+1 , ..., h n}. All items belonging to set {h i The vectors |i∈I} will be normalized by projection exponent to obtain the true label {x} corresponding to their predicted label [MASK]. i The probability distribution of |i∈I} across the entire model vocabulary. We optimize the predicted probability distribution of the model based on cross-entropy loss, making it more familiar with and understand the resume-job description corpus. The process is as follows: Figure 2 As shown.

[0033] 2. The contrastive learning phase, which is the phase of contrastive learning and training of the language model;

[0034] like Figure 1b As shown, we use weighted contrastive learning for training. Positive sample pairs are manually labeled resume-job description feature vector pairs at different recommendation stages from the headhunting company's talent recommendation records, while negative sample pairs are other samples from a randomly combined training batch. The training model aggregates positive sample pairs and alienates negative sample pairs in the implicit feature space.

[0035] Specifically, each batch of samples contains B resume-job feature vector pairs. And the corresponding B recommendation process states {s1, s2, ..., s B}. Wherein, for any resume feature vector Or job feature vector There is only one corresponding positive sample pair in this batch. or All the remaining feature vectors form negative sample pairs with it.

[0036] Each positive sample pair of feature vectors corresponds to a recommendation process state. Different recommendation processes assign different weights to the corresponding sample pairs in the contrastive learning loss calculation, thereby adjusting the impact of the sample pair on model training. For a given... and {s1, s2, ..., s} B The objective function for the comparative learning of this batch is:

[0037]

[0038] Here, we choose cosine distance to calculate the distance between two feature vectors, that is:

[0039]

[0040] The larger the value of dist(x, y), the closer the two eigenvectors x and y are in the vector space.

[0041] Wherein, the weight w(s) of the recommendation process state for any two resume-job description feature vector pairs is... i s j The choice of w(s) is diverse; in our implementation, we adopt a frequency-aware softweight strategy. This is because samples with higher recommendation rates (i.e., higher matching degrees) appear less frequently in the overall dataset, so we use w(s)... i s j The implementation strategy is as follows:

[0042] w(s i s j )=γf(s i f(s) j )

[0043]

[0044] We will include all of them in a batch and The corresponding soft weights are normalized to have a mean of one. Here, γ is a hyperparameter, and our setting follows the reference, being... Frequency-aware contrastive learning for neural machine translation .

[0045] Based on the above comparison objectives, we optimize the distribution of feature vectors corresponding to each resume and each job in the implicit space, so that the feature vectors corresponding to matching resumes and job positions are as close as possible in the space, while those that do not match are as far apart as possible.

[0046] 3. The verification and inference stage involves using the trained language model to recommend resumes;

[0047] like Figure 1c As shown, in the verification and reasoning phase, we first put the resume database... The N resume documents in the database are all represented as feature vectors by language models trained through contrastive learning, and stored as a resume vector library. Next, for job descriptions that require resume recommendations... JD Its feature vector h is also obtained through language modeling. JD Then, the cosine distance between the job vector and all resume vectors in the resume vector library in the implicit feature space is compared, that is, the following is calculated:

[0048]

[0049] We adopt the k-nearest neighbor approach, taking the k resume vectors in the feature space that are closest to the job vector as the result and returning them. That is:

[0050]

[0051] A Resume Recommendation Model Based on a Large-Scale Pre-trained Language Model. This invention designs a resume recommendation model based on a pre-trained language model. This model can further learn the text representation of resume-job related text using unlabeled resume and job description corpora. With a comparable amount of labeled data, it can better understand the text and learn better text feature vectors.

[0052] This invention employs a contrastive learning-based training approach. Drawing upon contrastive learning methods widely used in text representation learning, it adapts them for the first time to the resume recommendation domain. Furthermore, based on the characteristics of the resume-job training corpus, weighted training and other techniques are introduced during the training process to enhance the effectiveness of contrastive learning, thereby improving the accuracy of the model's resume recommendation.

[0053] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A resume recommendation method based on large-scale pre-trained language models and contrastive learning, characterized in that, include: Step S1: Use the pre-trained language model RoBERTa to obtain feature representation vectors for resume text and job description, which will output resume vector and job vector respectively; perform self-supervised pre-training on the model based on a large-scale resume-job description corpus to familiarize the model with the corpus. Step S2: Based on the resume vector and job vector, perform weighted contrastive learning training on the pre-trained language model; Step S3: Input the resumes in the resume library into the language model to obtain the resume vector library, and input the job descriptions that need to be recommended into the language model to obtain job vectors. Take the k resume vectors that are closest to the job vector in the feature space as recommendations, and return the results. In step S1, when using the pre-trained language model RoBERTa to obtain the feature representation vectors of the resume text and job description, the input is a plain text resume and job description. In step S1, during self-supervised pre-training, the masked language model is used to perform self-supervised pre-training on the resume-job description corpus of the language model. The input to the masked language model task is represented as a label sequence X = {x1, x2, ..., x...}. n Randomly sample the index set I∈{1,2,...,n}, and include all elements belonging to the set {x} i The label of |i∈I} is replaced with the label [MASK], resulting in a new label sequence X′={x1,[MASK],...x i-1 [MASK], x i+1 , ..., x n The model receives X' and represents it as an implicit state vector sequence H = {h1, h2, ..., h...}. i-1 h i h i+1 , ..., h n }; All belonging to set {h i The vectors |i∈I} will be projected exponent normalized to obtain the true label {x} corresponding to their predicted label [MASK]. i The probability distribution of |i∈I} over the entire model vocabulary; In step S1, during self-supervised pre-training, the probability distribution predicted by the model is optimized based on cross-entropy loss; In step S2, when performing weighted contrastive learning training, the positive sample pairs are the resume-job description feature vector pairs manually labeled in the headhunting company's talent recommendation records at different recommendation stages, and the negative sample pairs are other samples in the training batch that are randomly combined; and, when performing contrastive learning training, the language model aggregates positive sample pairs and alienates negative sample pairs in the implicit feature space. In step S2, each positive sample pair of resume-job description feature vector pairs corresponds to a recommendation process state, and different recommendation process states give different weights to the corresponding positive sample pairs in the comparison learning loss calculation. In step S2, the contrastive learning objective function used is: Where B represents the number of resume-job description feature vector pairs used for a batch comparison learning objective function. Let be the weighted contrastive learning loss of the resume vector of any number i in B. Let i be the weighted contrastive learning loss for the job vector of any number i in B.

2. The resume recommendation method based on large-scale pre-trained language models and contrastive learning according to claim 1, characterized in that: In step S2, the cosine distance is used to calculate the distance between the two feature vectors.

3. The resume recommendation method based on large-scale pre-trained language models and contrastive learning according to claim 2, characterized in that: In step S2, the weighting strategy for the recommendation process state of any two resume-job description feature vector pairs adopts a frequency-aware soft weight strategy.

4. The resume recommendation method based on large-scale pre-trained language models and contrastive learning according to any one of claims 1-3, characterized in that: In step S3, the cosine distance between the job vector and all vectors in the resume vector library in the implicit feature space is compared using a language model. The top k resume vectors that are closest to the job vector in the feature space are then selected as recommendations, and the results are returned.

Images