Resume classification method and device based on self-attention mechanism

Abstract

The present invention discloses a resume classification method and device based on a self-attention mechanism, the method comprising: obtaining resume information, performing text extraction on the resume information; performing information extraction on the pure text information after data cleaning to obtain work information; and classifying the work information using a convolutional neural network resume classification model. The convolutional neural network resume classification model comprises an embedding layer, a convolutional layer, a self-attention layer, a pooling layer, and a fully connected layer; the self-attention layer uses a self-attention mechanism to obtain long-distance dependency information for the local features, and further captures key classification features in the resume text based on the long-distance dependency information; the fully connected layer performs feature fusion on the output of the downsampled convolutional layer and the output of the self-attention layer and sends them to the Softmax function for classification to obtain the final classification result of the resume. The present application can effectively classify resumes, and has better classification performance and efficiency than other classification methods.

Description

Technical Field

[0001] The present invention relates to the technical field of text classification, and in particular to a resume classification method and device based on a self-attention mechanism. Background Art

[0002] With the rapid development of the Internet, electronic resume information has exploded. A resume is a formal document that job seekers use to present their work experience and skills to online recruitment websites or company human resources. In order to effectively analyze and manage resumes, improve the precise matching of talents and positions, and thus improve recruitment efficiency, one of the important links is to accurately classify resumes into industry categories. Traditional resume classification methods are mainly based on knowledge engineering and rely on manually constructed rules to classify resumes. This type of method cannot effectively cope with the massive resume files of different formats in the big data era.

[0003] The task of text classification refers to the process of automatically classifying a text set according to a certain classification system or rules. At present, text classification has become a hot research issue in the field of natural language processing. Resume text classification refers to the classification of job industry categories or job positions based on the job seeker's work skills and work experience in the resume text. Common job industry categories include information technology, education, finance, engineering, medical care, art, etc. Resume text classification technology realizes the automatic classification of resumes and provides an information source for subsequent talent recommendations.

[0004] Resume text classification can be achieved using text classification methods in natural language processing. Text classification usually includes text preprocessing, word segmentation, text representation, model construction and classification. The main text classification models based on machine learning include naive Bayes, K-nearest neighbor algorithm, logistic regression, decision tree and support vector machine. In recent years, thanks to the development of word vector technology and deep learning, text classification models based on deep learning have avoided the tedious problem of feature engineering construction based on machine learning methods, and achieved better performance in classification accuracy. Existing text classification methods based on deep learning usually embed the resume text first, then send it to models such as convolutional neural network (CNN) or recurrent neural network (RNN) for training, and finally use the trained model to predict the text category.

[0005] Classifying resume files by industry and position can provide support for subsequent resume recommendations. Resume classification methods based on natural language processing can be divided into unsupervised resume clustering and supervised resume classification according to whether there is a labeled data set. In the case of an unlabeled resume data set or an uncertain resume category, clustering algorithms such as K-means are usually used to cluster resumes by similarity. Unsupervised resume clustering can divide resumes into different categories based on similarity, but it is also necessary to combine manual analysis and category judgment of clustered resume data. In the case of a determined resume category, in order to classify resumes more accurately, a supervised resume classification model is used to predict the category of unknown resumes. Resume classification based on machine learning requires feature engineering for feature extraction and feature selection, while text classification models based on deep learning can automatically extract text features. In recent years, with the rise of deep learning in natural language processing, more and more resume classification studies based on deep learning have been reported. Existing methods still need to be improved in improving the accuracy of resume classification. Summary of the invention

[0006] In view of the problem that the accuracy of existing resume classification methods needs to be improved, this application takes into account the different contributions of the features composed of different words in the resume to resume classification, introduces the self-attention mechanism into the classic convolutional neural network text classification model, and expresses the resume text features more richly. The resume dataset was used for experimental verification, and the experimental results show that the CNN-Attention model helps to further improve the accuracy of resume classification. This application introduces the self-attention mechanism into the classification model, assigns corresponding weights according to the influence of the features composed of different words on the classification effect, so that the classification model can further extract important features in the resume text and achieve more accurate resume classification.

[0007] The present application discloses a resume classification method based on a self-attention mechanism, including obtaining resume information, performing text extraction on the resume information to obtain plain text information, performing data cleaning on the plain text information, performing information extraction on the plain text information after data cleaning to obtain job information, and classifying the job information using a convolutional neural network resume classification model.

[0008] The data cleaning of the plain text information includes removing special symbols, non-printing characters, extra blank lines and basic personal information in the text.

[0009] The data cleaning of the plain text information also includes data smoothing processing of the obtained plain text information to eliminate the collection errors that occur within a certain period of time; specifically, the plain text information obtained for the same resume within a period of time is segmented, and each segment constitutes a text vector yi , the N text vectors obtained within a period of time are combined into a text vector group [y1,y2,…,y N ], calculate the mutual correlation matrix C of the text vector group, perform eigenvalue decomposition on the mutual correlation matrix C, and obtain:

[0010] C=VDV H ,

[0011] Among them, V is the eigenvector matrix, D is the eigenvalue matrix, the diagonal elements of the matrix D are normalized and used as weight vectors, and the text vector groups collected over a period of time are weighted summed to obtain the smoothed value of the pure text information obtained for the same resume over a period of time as the cleaned data.

[0012] The work information includes work skill information and work experience information.

[0013] The convolutional neural network resume classification model includes an embedding layer, a convolutional layer, a self-attention layer, a pooling layer, and a fully connected layer. The embedding layer obtains the corresponding word embedding vector according to the working information of the input convolutional neural network resume classification model, and inputs it into the convolutional layer; the convolutional layer performs a one-dimensional convolution operation on the word embedding vector to extract its local features, and inputs the local features into the self-attention layer; the self-attention layer uses the self-attention mechanism to obtain long-distance dependency information for the local features, and further captures the key classification features in the resume text based on the long-distance dependency information. The pooling layer uses the maximum pooling operation to downsample the outputs of the self-attention layer and the convolutional layer respectively, and obtains two downsampled results, and the two downsampled results are sent to the fully connected layer, and the fully connected layer performs feature fusion on the two downsampled results and sends them to the Softmax function for classification to obtain the final classification result of the resume.

[0014] The embedding layer uses the trained continuous bag of words model in Word2Vec to obtain the word embedding vector of each word in the work information, and uses the word embedding vector to form a matrix, which is used as the input of the convolution layer. The matrix formed by the word embedding vector is represented as a two-dimensional matrix X of n×d dimensions, and its expression is:

[0015] X=[x1,x2,…x i ,…,x n ] T ∈R n×d ,

[0016] Where n is the number of word embedding vectors contained in the work information, d is the word embedding vector dimension, and x i is the word embedding vector of the i-th word in the job information.

[0017] The fused feature vector is sent to the Softmax function for classification, and its calculation formula is:

[0018] y i =softmax(∑W s ·V+b s ),

[0019] where y i Represents the category obtained after classification prediction, W s is the weight matrix of the fully connected layer, V is the feature vector after feature fusion, b s is the bias of the fully connected layer.

[0020] The self-attention layer first maps the text sequence contained in the local features into query variables Q, key-value variables K and value variables V, then performs a dot product operation on the query variable Q and the key-value variable K, and then normalizes the dot product operation result. The normalized value is used as the weight coefficient of the value variable. Finally, the weight coefficient and the value variable are weighted and summed, and then processed with the softmax function to obtain the output of the self-attention layer.

[0021] The text sequence contained in the local feature is mapped into the query variable Q, the key variable K and the value variable V, and the specific calculation process is as follows:

[0022] Q=SW Q ,K=SW K ,V=SW V ,

[0023] Among them, W Q , W K , W V are the linear transformation matrices of the query variable Q, key variable K and value variable V respectively, and S is the text sequence matrix contained in the local features.

[0024] The resume classification model of this application uses cross entropy as the loss function of the model. The calculation formula is as follows, where Loss represents the loss function, n is the number of resume samples, m is the number of resume categories, and p is ik represents the probability that the i-th resume sample belongs to category k, y ik It represents the probability that the resume classification model predicts the i-th sample as category k:

[0025]

[0026] During model training, the performance of the convolutional neural network resume classification model is monitored by using the changes in the loss function values ​​of the resume classification model on the resume text training set and validation set. After each round of training, the loss values ​​of the resume classification model on the training set and validation set are calculated, and the model parameters are updated at the same time. After the training starts, the loss values ​​of the resume classification model on the training set and validation set will continue to decrease. When the loss value of the resume classification model on the validation set is greater than the previous training result, the training is stopped. At this time, it means that the resume classification model has been overfitted, and the parameters in the previous iteration result are used as the final parameters of the model.

[0027] The performance of the convolutional neural network resume classification model is monitored using the resume text test data, label data, and training data. When abnormal performance is detected, the parameters of the convolutional neural network resume classification model are updated, including:

[0028] The performance of the convolutional neural network resume classification model is monitored by distinguishing the differences between the test data, the label data and the training data; the data of the resume text is regarded as a stationary random process, and corresponding autoregressive-moving average models, namely, ARMA models, are established for the test data, the label data and the training data, respectively, to obtain the first ARMA model, the second ARMA model and the third ARMA model, respectively, and the coefficients of the three ARMA models are calculated. The cross-correlation matrix is ​​calculated to obtain the maximum eigenvalue, and the maximum eigenvalue is used to distinguish the differences between the test data, the label data and the training data; when the maximum eigenvalue is greater than the difference discrimination threshold, it is determined that there is a significant difference between the test data, the label data and the training data, and then it is considered that the performance of the convolutional neural network resume classification model is abnormal, the convolutional neural network resume classification model is trained using the label data and the training data, and the parameters of the convolutional neural network resume classification model are updated according to the training results until the performance of the convolutional neural network resume classification model is no longer abnormal, and the training of the convolutional neural network resume classification model is stopped.

[0029] The present application discloses a resume classification device based on a self-attention mechanism, including a resume information extraction module and a convolutional neural network resume classification model. The resume information extraction module is used to obtain resume information, perform text extraction on the resume information to obtain plain text information, perform data cleaning on the plain text information, perform information extraction on the plain text information after data cleaning to obtain job information, and output the job information to the convolutional neural network resume classification model; the convolutional neural network resume classification model is used to classify the job information and obtain a resume classification result.

[0030] The data cleaning of the plain text information includes removing special symbols, non-printing characters, extra blank lines and basic personal information in the text.

[0031] The data cleaning of the plain text information also includes data smoothing processing of the obtained plain text information to eliminate the collection errors that occur within a certain period of time; specifically, the plain text information obtained for the same resume within a period of time is segmented, and each segment constitutes a text vector y i , the N text vectors obtained within a period of time are combined into a text vector group [y1,y2,…,y N ], calculate the mutual correlation matrix C of the text vector group, perform eigenvalue decomposition on the mutual correlation matrix C, and obtain:

[0032] C=VDV H ,

[0033] Among them, V is the eigenvector matrix, D is the eigenvalue matrix, the diagonal elements of the matrix D are normalized and used as weight vectors, and the text vector groups collected over a period of time are weighted summed to obtain the smoothed value of the pure text information obtained for the same resume over a period of time as the cleaned data.

[0034] The convolutional neural network resume classification model includes an embedding layer, a convolutional layer, a self-attention layer, a pooling layer and a fully connected layer. The embedding layer obtains the corresponding word embedding vector according to the working information of the input convolutional neural network resume classification model, and inputs it into the convolutional layer; the convolutional layer performs a one-dimensional convolution operation on the word embedding vector to extract its local features, and inputs the local features into the self-attention layer; the self-attention layer uses the self-attention mechanism to obtain long-distance dependency information for the local features, and further captures the key classification features in the resume text based on the long-distance dependency information. The pooling layer uses the maximum pooling operation to downsample the outputs of the self-attention layer and the convolutional layer respectively, obtains two downsampled results, and sends the two downsampled results to the fully connected layer, the fully connected layer performs feature fusion on the two downsampled results and sends them to the Softmax function for classification, and obtains the final classification result of the resume.

[0035] The embedding layer uses the trained continuous bag of words model in Word2Vec to obtain the word embedding vector of each word in the work information, and uses the word embedding vector to form a matrix, which is used as the input of the convolution layer. The matrix formed by the word embedding vector is represented as a two-dimensional matrix X of n×d dimensions, and its expression is:

[0036] X=[x1,x2,…x i ,…,x n ]T ∈R n×d ,

[0037] Where n is the number of word embedding vectors contained in the work information, d is the word embedding vector dimension, and x i is the word embedding vector of the i-th word in the job information.

[0038] The fused feature vector is sent to the Softmax function for classification, and its calculation formula is:

[0039] y i =softmax(∑W s ·V+b s ),

[0040] where y i Represents the category obtained after classification prediction, W s is the weight matrix of the fully connected layer, V is the feature vector after feature fusion, b s is the bias of the fully connected layer.

[0041] The self-attention layer first maps the text sequence contained in the local features into query variables Q, key-value variables K and value variables V, then performs a dot product operation on the query variable Q and the key-value variable K, and then normalizes the dot product operation result. The normalized value is used as the weight coefficient of the value variable. Finally, the weight coefficient and the value variable are weighted and summed, and then processed with the softmax function to obtain the output of the self-attention layer.

[0042] The text sequence contained in the local feature is mapped into the query variable Q, the key variable K and the value variable V, and the specific calculation process is as follows:

[0043] Q=SW Q ,K=SW K ,V=SW V ,

[0044] Among them, W Q , W K , W V are the linear transformation matrices of the query variable Q, key variable K and value variable V respectively, and S is the text sequence matrix contained in the local features.

[0045] The resume classification model of this application uses cross entropy as the loss function of the model. The calculation formula is as follows, where Loss represents the loss function, n is the number of resume samples, m is the number of resume categories, and p is ik represents the probability that the i-th resume sample belongs to category k, y ikIt represents the probability that the resume classification model predicts the i-th sample as category k:

[0046]

[0047] During model training, the performance of the convolutional neural network resume classification model is monitored by using the changes in the loss function values ​​of the resume classification model on the resume text training set and validation set. After each round of training, the loss values ​​of the resume classification model on the training set and validation set are calculated, and the model parameters are updated at the same time. After the training starts, the loss values ​​of the resume classification model on the training set and validation set will continue to decrease. When the loss value of the resume classification model on the validation set is greater than the previous training result, the training is stopped. At this time, it means that the resume classification model has been overfitted, and the parameters in the previous iteration result are used as the final parameters of the model.

[0048] The performance of the convolutional neural network resume classification model is monitored using the resume text test data, label data, and training data. When abnormal performance is detected, the parameters of the convolutional neural network resume classification model are updated, including:

[0049] The performance of the convolutional neural network resume classification model is monitored by distinguishing the differences between the test data, the label data and the training data; the data of the resume text is regarded as a stationary random process, and corresponding autoregressive-moving average models, namely, ARMA models, are established for the test data, the label data and the training data, respectively, to obtain the first ARMA model, the second ARMA model and the third ARMA model, respectively, and the coefficients of the three ARMA models are calculated. The cross-correlation matrix is ​​calculated to obtain the maximum eigenvalue, and the maximum eigenvalue is used to distinguish the differences between the test data, the label data and the training data; when the maximum eigenvalue is greater than the difference discrimination threshold, it is determined that there is a significant difference between the test data, the label data and the training data, and then it is considered that the performance of the convolutional neural network resume classification model is abnormal, the convolutional neural network resume classification model is trained using the label data and the training data, and the parameters of the convolutional neural network resume classification model are updated according to the training results until the performance of the convolutional neural network resume classification model is no longer abnormal, and the training of the convolutional neural network resume classification model is stopped.

[0050] The beneficial effects of the present invention are:

[0051] This application can effectively classify resumes and has better classification performance and classification efficiency than other classification methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1It is the overall implementation flow chart of the method of the present invention;

[0053] Figure 2 It is a basic composition diagram of the convolutional neural network resume classification model of the present invention;

[0054] Figure 3 It is a structural diagram of the convolutional layer composition of the present invention;

[0055] Figure 4 It is the one-dimensional convolution operation process of the present invention (the step size is 1);

[0056] Figure 5 Schematic diagram of the calculation process of the self-attention layer of the present invention;

[0057] Figure 6 Schematic diagram of the training of four models;

[0058] Figure 7 The following is a graph showing the performance of the four models on the test set. DETAILED DESCRIPTION

[0059] In order to better understand the content of the present invention, an embodiment is given here. Figure 1 It is the overall implementation flow chart of the method of the present invention; Figure 2 It is a basic composition diagram of the convolutional neural network resume classification model of the present invention; Figure 3 It is a structural diagram of the convolutional layer composition of the present invention; Figure 4 It is the one-dimensional convolution operation process of the present invention (the step size is 1); Figure 5 Schematic diagram of the calculation process of the self-attention layer of the present invention; Figure 6 Schematic diagram of the training of four models; Figure 7 The following is a graph showing the performance of the four models on the test set.

[0060] This application discloses a resume classification method based on the self-attention mechanism. The overall framework of the resume classification scheme proposed in this application is as follows: Figure 1As shown. The electronic resumes submitted by job seekers are in various formats, such as PDF, DOC, DOCX, etc. Therefore, before classifying the resumes, it is necessary to capture the plain text information from the electronic resumes in different formats; then perform data cleaning on the resume text, such as removing special symbols, non-printing characters, and extra blank lines in the text. Considering that personal basic information in the resume, such as email, phone number, personal hobbies and other attributes, has no effect on the results of resume classification, this application regards this information as interference information and does not consider it, and only cares about the work skills and work experience information in the resume. This application adds an information extraction module before resume classification, which is responsible for extracting work experience and work skills information from the cleaned resume text, and then sends the above-mentioned extracted text information to the CNN-Attention resume classification model proposed in this application to classify the resume.

[0061] The present application discloses a resume classification method based on a self-attention mechanism, including obtaining resume information, performing text extraction on the resume information to obtain plain text information, performing data cleaning on the plain text information, performing information extraction on the plain text information after data cleaning to obtain job information, and classifying the job information using a convolutional neural network resume classification model.

[0062] The data cleaning of the plain text information includes removing special symbols, non-printing characters, extra blank lines and basic personal information in the text.

[0063] The data cleaning of the plain text information also includes data smoothing processing of the obtained plain text information to eliminate the collection errors that occur within a certain period of time; specifically, the plain text information obtained for the same resume within a period of time is segmented, and each segment constitutes a text vector y i , the N text vectors obtained within a period of time are formed into a text vector group [y1, y2, ..., yN], the cross-correlation matrix C of the text vector group is calculated, and the eigenvalue decomposition of the cross-correlation matrix C is performed to obtain:

[0064] C=VDV H ,

[0065] Among them, V is the eigenvector matrix, D is the eigenvalue matrix, the diagonal elements of the matrix D are normalized and used as weight vectors, and the text vector groups collected over a period of time are weighted summed to obtain the smoothed value of the pure text information obtained for the same resume over a period of time as the cleaned data.

[0066] The work information includes work skill information and work experience information.

[0067] The convolutional neural network resume classification model includes an embedding layer, a convolution layer, a self-attention layer, a pooling layer, and a fully connected layer. The embedding layer obtains the corresponding word embedding vector according to the working information of the input convolutional neural network resume classification model, and inputs it into the convolution layer; the convolution layer performs a one-dimensional convolution operation on the word embedding vector to extract its local features, and inputs the local features into the self-attention layer; the self-attention layer uses the self-attention mechanism to obtain long-distance dependency information for the local features, and further captures the key classification features in the resume text based on the long-distance dependency information. The pooling layer uses the maximum pooling operation to downsample the outputs of the self-attention layer and the convolution layer, and sends the two downsampled results to the fully connected layer. The fully connected layer performs feature fusion on the output of the downsampled convolution layer and the output of the self-attention layer and sends them to the Softmax function for classification to obtain the final classification result of the resume.

[0068] The method obtains the corresponding word embedding vector based on the working information of the input convolutional neural network resume classification model, and uses the continuous bag of words model in Word2Vec to implement it.

[0069] For resume files of different formats, the two tools doctotext and pdftotext are used to extract text from PDF, DOC, and DOCX. After obtaining the resume text application, the resume text was cleaned to remove unprintable characters. Information such as name, email address, phone number, and graduation school in the resume is not helpful for the classification results, but may reduce the accuracy of the classification. At the same time, it also increases the computational complexity of the classification model. Therefore, this application finally determined the scheme of using work skills and work experience information after information extraction to classify resumes. Resume information extraction can be implemented using rule-based or statistical methods.

[0070] The embedding layer uses the trained continuous bag of words model in Word2Vec to obtain the word embedding vector of each word in the work information, and uses the word embedding vector to form a matrix, which is used as the input of the convolution layer. The matrix formed by the word embedding vector is represented as a two-dimensional matrix X of n×d dimensions, and its expression is:

[0071] X=[x1,x2,…x i ,…,x n ] T ∈R n×d ,

[0072] Where n is the number of word embedding vectors contained in the work information, d is the word embedding vector dimension, and x i is the word embedding vector of the i-th word in the job information.

[0073] Convolutional neural network is a kind of neural network with characteristics such as local connection and weight sharing. It has achieved good results in fields such as computer vision and natural language processing. Convolutional neural network is usually composed of input layer, convolution layer, pooling layer, fully connected layer and output layer. Convolutional neural network has two major characteristics: local perception and parameter sharing. Local perception means that each neuron of convolutional neural network does not need to perceive all the information of input text data, but only perceives the local information of input text data, and then merges these local information at a higher level to obtain the global representation information of resume text. The network structure with weight sharing reduces the number of weights and reduces the complexity of the network model. Convolutional neural network uses word embedding of resume text as input, which can effectively learn the corresponding features from resume text and avoid the complex feature extraction process. The composition structure of convolutional layer is as follows: Figure 3 As shown in the figure, a total of three sub-convolutional layers are designed, the size of the convolution kernel is 3×100, and the number of convolution kernels of the three sub-convolutional layers are 264, 128, and 64 respectively.

[0074] This application uses one-dimensional convolution to process the n×d-dimensional resume text data output by the embedding layer. The one-dimensional convolution operation process is as follows: Figure 4 As shown in the figure, the width of the convolution kernel in the one-dimensional convolution is equal to the dimension of the word vector, and the convolution kernel moves along the sequence order of the resume text. The one-dimensional convolution operation of CNN is as follows:

[0075] c i =f(∑W c ·X i:i+h-1 +b c ),

[0076] C=[c1,c2,…c i ,…c n-h+1 ],

[0077] Among them, c i represents the result after the convolution operation, that is, the activation output after the dot product of the input word vector matrix and the convolution kernel and the addition of the bias. f represents the activation function. The activation function f used in this application is the ReLU function. W c represents the weight matrix of the convolution kernel, h represents the width of the convolution kernel, X i:i+h-1 represents the word vector matrix within the i-th to i+h-1-th window of the input, b c The output of the convolution layer is C. After obtaining local features through three layers of convolution calculation, the pooling layer is used to compress the extracted local features in order to reduce the complexity of network calculation. The pooling layer uses maximum pooling. The outputs of the convolution layer and the self-attention layer after pooling are concatenated to obtain a 128×1 one-dimensional feature vector V. The calculation formula is as follows:

[0078]

[0079] in, represents the output of the convolutional layer after maximum pooling, Represents the output of the self-attention layer after the maximum pooling. Finally, the feature vector after feature fusion is fully connected with the category neuron for classification, and the resume category is predicted through the softmax function.

[0080] The fused feature vector is sent to the Softmax function for classification, and its calculation formula is:

[0081] y i =softmax(∑W s ·V+b s ),

[0082] where y i Represents the category obtained after classification prediction, W s is the weight matrix of the fully connected layer, V is the feature vector after feature fusion, b s is the bias of the fully connected layer. After using the convolutional neural network to extract features from the resume text, this application adds a self-attention mechanism to further extract classification features from the resume text. The self-attention layer is responsible for extracting long-distance dependency information from the resume text. The attention mechanism is derived from the human visual attention mechanism, which can extract and focus on important classification features in the resume text. Therefore, applying the attention mechanism to the resume classification task can further extract important features of the resume text. The calculation process of the self-attention layer is as follows: Figure 5 As shown. The self-attention layer first maps the text sequence contained in the local features into query variables Q, key variables K and value variables V, then performs a dot product operation on the query variable Q and the key variable K, and then normalizes the dot product operation result, and uses the normalized value as the weight coefficient of the value variable. Finally, the weight coefficient and the value variable are weighted and summed, and then processed with the softmax function to obtain the output of the self-attention layer. The specific calculation process of mapping the text sequence contained in the local features into query variables Q, key variables K and value variables V is as follows:

[0083] Q=SW Q ,K=SW K ,V=SW V ,

[0084] Among them, W Q , W K , W Vare the linear transformation matrices of the query variable Q, key variable K, and value variable V, respectively, and S is the text sequence matrix contained in the local features. By adopting the self-attention mechanism, the present application reduces the dependence on external information and is better at capturing the internal correlation of data or features. The self-attention mechanism applied in the resume text can fully capture the long-distance dependencies of the internal data of the resume text sequence and assign different weights to different attributes. The self-attention mechanism of the present application maps the output C of the convolutional layer to a set of vector outputs of query Q, key K, and value V. The calculation formula of the output H of the self-attention layer is shown below,

[0085]

[0086] Among them, d k Represents the dimension of K. Cross entropy is used as the loss function of the model, and its calculation formula is as follows, where Loss represents the loss function, n is the number of resume samples, m is the number of resume categories, and t ik represents the probability that the i-th resume sample belongs to category k, p ik It represents the probability that the resume classification model predicts the i-th sample as category k:

[0087]

[0088] During model training, the performance of the convolutional neural network resume classification model is monitored by using the changes in the loss function values ​​of the resume classification model on the resume text training set and validation set. After each round of training, the loss values ​​of the resume classification model on the training set and validation set are calculated, and the model parameters are updated at the same time. After the training starts, the loss values ​​of the resume classification model on the training set and validation set will continue to decrease. When the loss value of the resume classification model on the validation set is greater than the previous training result, the training is stopped. At this time, it means that the resume classification model has been overfitted, and the parameters in the previous iteration result are used as the final parameters of the model.

[0089] The performance of the convolutional neural network resume classification model is monitored using the test data, label data and training data of the resume text. When the performance is abnormal, the parameters of the convolutional neural network resume classification model are updated, including: distinguishing the differences between the test data, label data and training data, so as to realize the performance monitoring of the convolutional neural network resume classification model; considering the data of the resume text as a stationary random process, establishing corresponding autoregressive-moving average models, namely ARMA models, for the test data, label data and training data, respectively, and obtaining the first ARMA model, the second ARMA model and the third ARMA model respectively, and calculating the coefficients of the three ARMA models. Calculate its mutual correlation matrix, calculate the maximum eigenvalue of the mutual correlation matrix, and use the maximum eigenvalue to distinguish the differences between the test data, label data and training data; when the maximum eigenvalue is greater than the difference judgment threshold, it is determined that there is a significant difference between the test data, label data and training data, and then it is considered that the performance of the convolutional neural network resume classification model is abnormal, and the convolutional neural network resume classification model is trained using the label data and the training data, and the parameters of the convolutional neural network resume classification model are updated according to the training results until the performance of the convolutional neural network resume classification model is no longer monitored to be abnormal, and the training of the convolutional neural network resume classification model is stopped.

[0090] The present application discloses a resume classification device based on a self-attention mechanism, including a resume information extraction module and a convolutional neural network resume classification model. The resume information extraction module is used to obtain resume information, perform text extraction on the resume information to obtain plain text information, perform data cleaning on the plain text information, perform information extraction on the plain text information after data cleaning to obtain job information, and output the job information to the convolutional neural network resume classification model; the convolutional neural network resume classification model is used to classify the job information and obtain a resume classification result.

[0091] The data cleaning of the plain text information includes removing special symbols, non-printing characters, extra blank lines and basic personal information in the text.

[0092] The data cleaning of the plain text information also includes data smoothing processing of the obtained plain text information to eliminate the collection errors that occur within a certain period of time; specifically, the plain text information obtained for the same resume within a period of time is segmented, and each segment constitutes a text vector y i , the N text vectors obtained within a period of time are combined into a text vector group [y1,y2,…,y N ], calculate the mutual correlation matrix C of the text vector group, perform eigenvalue decomposition on the mutual correlation matrix C, and obtain:

[0093] C=VDV H ,

[0094] Among them, V is the eigenvector matrix, D is the eigenvalue matrix, the diagonal elements of the matrix D are normalized and used as weight vectors, and the text vector groups collected over a period of time are weighted summed to obtain the smoothed value of the pure text information obtained for the same resume over a period of time as the cleaned data.

[0095] The convolutional neural network resume classification model includes an embedding layer, a convolutional layer, a self-attention layer, a pooling layer and a fully connected layer. The embedding layer obtains the corresponding word embedding vector according to the working information of the input convolutional neural network resume classification model, and inputs it into the convolutional layer; the convolutional layer performs a one-dimensional convolution operation on the word embedding vector to extract its local features, and inputs the local features into the self-attention layer; the self-attention layer uses the self-attention mechanism to obtain long-distance dependency information for the local features, and further captures the key classification features in the resume text based on the long-distance dependency information. The pooling layer uses the maximum pooling operation to downsample the outputs of the self-attention layer and the convolutional layer, and sends the two results obtained by downsampling to the fully connected layer. The fully connected layer performs feature fusion on the output of the downsampled convolutional layer and the output of the self-attention layer and sends them to the Softmax function for classification to obtain the final classification result of the resume.

[0096] The embedding layer uses the trained continuous bag of words model in Word2Vec to obtain the word embedding vector of each word in the work information, and uses the word embedding vector to form a matrix, which is used as the input of the convolution layer. The matrix formed by the word embedding vector is represented as a two-dimensional matrix X of n×d dimensions, and its expression is:

[0097] X=[x1,x2,…x i ,…,x n ] T ∈R n×d ,

[0098] Where n is the number of word embedding vectors contained in the work information, d is the word embedding vector dimension, and x i is the word embedding vector of the i-th word in the job information.

[0099] The fused feature vector is sent to the Softmax function for classification, and its calculation formula is:

[0100] y i =softmax(∑W s ·V+b s ),

[0101] where y i Represents the category obtained after classification prediction, W s is the weight matrix of the fully connected layer, V is the feature vector after feature fusion, b s is the bias of the fully connected layer.

[0102] The self-attention layer first maps the text sequence contained in the local features into query variables Q, key-value variables K and value variables V, then performs a dot product operation on the query variable Q and the key-value variable K, and then normalizes the dot product operation result. The normalized value is used as the weight coefficient of the value variable. Finally, the weight coefficient and the value variable are weighted and summed, and then processed with the softmax function to obtain the output of the self-attention layer.

[0103] The text sequence contained in the local feature is mapped into the query variable Q, the key variable K and the value variable V, and the specific calculation process is as follows:

[0104] Q=SW Q ,K=SW K ,V=SW V ,

[0105] Among them, W Q , W K , W V are the linear transformation matrices of the query variable Q, key variable K and value variable V respectively, and S is the text sequence matrix contained in the local features.

[0106] The resume classification model of this application uses cross entropy as the loss function of the model. The calculation formula is as follows, where Loss represents the loss function, n is the number of resume samples, m is the number of resume categories, and p is ik represents the probability that the i-th resume sample belongs to category k, y ik It represents the probability that the resume classification model predicts the i-th sample as category k:

[0107]

[0108] During model training, the performance of the convolutional neural network resume classification model is monitored by using the changes in the loss function values ​​of the resume classification model on the resume text training set and validation set. After each round of training, the loss values ​​of the resume classification model on the training set and validation set are calculated, and the model parameters are updated at the same time. After the training starts, the loss values ​​of the resume classification model on the training set and validation set will continue to decrease. When the loss value of the resume classification model on the validation set is greater than the previous training result, the training is stopped. At this time, it means that the resume classification model has been overfitted, and the parameters in the previous iteration result are used as the final parameters of the model.

[0109] The performance of the convolutional neural network resume classification model is monitored using the resume text test data, label data, and training data. When abnormal performance is detected, the parameters of the convolutional neural network resume classification model are updated, including:

[0110] The performance of the convolutional neural network resume classification model is monitored by distinguishing the differences between the test data, the label data and the training data; the data of the resume text is regarded as a stationary random process, and corresponding autoregressive-moving average models, namely, ARMA models, are established for the test data, the label data and the training data, respectively, to obtain the first ARMA model, the second ARMA model and the third ARMA model, respectively, and the coefficients of the three ARMA models are calculated. The cross-correlation matrix is ​​calculated to obtain the maximum eigenvalue, and the maximum eigenvalue is used to distinguish the differences between the test data, the label data and the training data; when the maximum eigenvalue is greater than the difference discrimination threshold, it is determined that there is a significant difference between the test data, the label data and the training data, and then it is considered that the performance of the convolutional neural network resume classification model is abnormal, the convolutional neural network resume classification model is trained using the label data and the training data, and the parameters of the convolutional neural network resume classification model are updated according to the training results until the performance of the convolutional neural network resume classification model is no longer abnormal, and the training of the convolutional neural network resume classification model is stopped.

[0111] In order to classify resumes, it is necessary to use the resume corpus with annotated categories to train and test the resume classification model. The resume dataset in this application research comes from the public resume dataset provided on the Kaggle platform. The statistics of the number of resumes in each category of the dataset are shown in Table 1. There are 25 annotated categories, including Data Science, HR, Advocate, Arts, etc. The statistics of the resume dataset categories are shown in Table 1.

[0112] Table 1 Statistics of resume dataset categories

[0113]

[0114] This application selects three indicators, namely precision, recall and F1-score, to evaluate the performance of the model, and their definitions are shown in Table 2. Among them, TP is the number of resumes whose resume categories are correctly identified by the resume classification model, FP is the number of irrelevant resumes that are incorrectly identified by the resume classification model, and FN is the number of resumes of related categories that the resume classification model fails to correctly identify.

[0115] Table 2 Definition of evaluation indicators

[0116]

[0117] This application uses Python 3.7 and TensorFlow to build the model, which is completed under the 64-bit Ubuntu 20.04 operating system. The computer configuration is an 8-core 16-thread CPU, 16GB memory, and the graphics card is NVIDIA GeForce RTX 2080. When training and evaluating the model, the ratio of the number of training sets, validation sets, and test sets is 6:2:2. This application first cleans the collected resume data set, including removing line breaks and some special interference characters, and then sends the resume data to the classification model for training. In order to compare and evaluate the performance of the CNN-Attention resume classification model, this application uses the resume data set to conduct comparative experiments on the following 4 models and observe the evaluation index results of the experiment. 1) BiLSTM model: After the resume text is embedded in the word, it is sent to the bidirectional long short-term memory neural network for resume classification. 2) BiGRU model: After the resume text is embedded in the word, it is sent to the bidirectional gate control unit neural network for resume classification. 3) CNN model: After the resume text is embedded in the word, it is sent to the convolutional neural network for resume classification. 4) Model of this application: that is, the resume classification model combined with the self-attention mechanism proposed in this application.

[0118] The word vector dimension obtained by this application through Word2vec is set to 100. According to the statistics of the text length of the resume data, the sequence length of the model is set to 1148. The excess is truncated and the insufficient is filled with zero. The input of the convolutional layer is a 1148×100 dimensional vector matrix. Batch_size is 64, epoch is 25, the optimizer is Adam, and the initial learning rate is 0.001. This application uses dropout to avoid overfitting. The principle of dropout is to randomly discard some node data of the neural network during training, and then use all nodes when testing the model. Dropout is set to 0.4.

[0119] Figure 6The training results of the four models are shown in 25 epochs. Figure 6 It can be seen that in the two classification models based on recurrent neural networks, the performance of the BiGRU model on the training set is better than that of the BiLSTM model. In the two models that use convolutional neural networks for feature extraction, the performance of the CNN-Attention model combined with the attention mechanism is better than that of the CNN model that only uses convolutional neural networks. In the 15th round of training, the accuracy of the CNN-Attention model on the training set reached 100%, indicating that adding the self-attention mechanism can further improve the performance of the resume classification model. Figure 7 The F1-score performance of the four models on the resume test set is shown, from which it can be seen that the classification accuracy of the two resume classification models based on convolutional neural networks is higher than that of the BiLSTM and BiGRU models.

[0120] Table 3 shows the evaluation index results of the four models on the test set under the same experimental environment conditions and training parameters. As can be seen from Table 3, after 25 rounds of training, the precision of the CNN-Attention resume classification model is 97.61%, the recall rate is 97.14%, and the F1 value is 97.26%. All indicators have reached the highest level. The comparative experiment shows that when establishing a resume classification model, the CNN-Attention model combined with the self-attention mechanism has stronger feature extraction capabilities and higher classification accuracy than other models.

[0121] Table 3 Comparative experimental results (unit: %)

[0122]

[0123] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A resume classification method based on self-attention mechanism, characterized in that: include: Obtain resume information, perform text extraction on the resume information, and obtain plain text information; Perform data cleaning on plain text information; Extract information from the plain text information after data cleaning to obtain work information; The job information is classified using a convolutional neural network resume classification model; The data cleaning of the plain text information includes removing special symbols, non-printing characters, extra blank lines and basic personal information in the text; The data cleaning of the plain text information also includes data smoothing processing of the obtained plain text information to eliminate the collection errors that occur within a certain period of time; specifically, the plain text information obtained for the same resume within a period of time is segmented, and each segment constitutes a text vector y i , the N text vectors obtained within a period of time are combined into a text vector group [y1,y2,…,y N ], calculate the mutual correlation matrix C of the text vector group, perform eigenvalue decomposition on the mutual correlation matrix C, and obtain: C=VDV H , Among them, V is the eigenvector matrix, D is the eigenvalue matrix, the diagonal elements of the matrix D are normalized and used as weight vectors, and the text vector groups collected within a period of time are weighted summed to obtain the smoothed value of the pure text information obtained for the same resume within a period of time as the cleaned data; The convolutional neural network resume classification model includes an embedding layer, a convolutional layer, a self-attention layer, a pooling layer and a fully connected layer; the embedding layer obtains the corresponding word embedding vector according to the working information of the input convolutional neural network resume classification model, and inputs it into the convolutional layer; the convolutional layer performs a one-dimensional convolution operation on the word embedding vector to extract its local features, and inputs the local features into the self-attention layer; the self-attention layer uses the self-attention mechanism to obtain long-distance dependency information for the local features, and further captures the key classification features in the resume text according to the long-distance dependency information; the pooling layer uses the maximum pooling operation to downsample the outputs of the self-attention layer and the convolutional layer respectively, obtains two downsampled results, and sends the two downsampled results to the fully connected layer, the fully connected layer performs feature fusion on the two downsampled results and sends them to the Softmax function for classification, and obtains the final classification result of the resume; The embedding layer uses the trained continuous bag-of-words model in Word2Vec to obtain the word embedding vector of each word in the work information, uses the word embedding vector to form a matrix, and uses it as the input of the convolutional layer; the matrix formed by the word embedding vector is represented as a two-dimensional matrix X of n×d dimensions, and its expression is: X=[x1,x2,···x i ,···,x n ] T ∈R n×d , Where n is the number of word embedding vectors contained in the work information, d is the word embedding vector dimension, and x i is the word embedding vector of the i-th word in the job information.

2. The resume classification method based on the self-attention mechanism as claimed in claim 1, characterized in that: include: The work information includes work skill information and work experience information.

3. The resume classification method based on the self-attention mechanism as claimed in claim 1, characterized in that: include: The fused feature vector is sent to the Softmax function for classification, and its calculation formula is: y i =softmax(∑W s ·V+b s ), where y i Represents the category obtained after classification prediction, W s is the weight matrix of the fully connected layer, V is the feature vector after feature fusion, b s is the bias of the fully connected layer.

4. The resume classification method based on the self-attention mechanism as claimed in claim 1, characterized in that: include: The self-attention layer first maps the text sequence contained in the local features into a query variable Q, a key variable K and a value variable V, then performs a dot product operation on the query variable Q and the key variable K, and then normalizes the dot product operation result, and uses the normalized value as the weight coefficient of the value variable. Finally, the weight coefficient and the value variable are weighted and summed, and then processed with a softmax function to obtain the output of the self-attention layer.

5. The resume classification method based on the self-attention mechanism as claimed in claim 1, characterized in that: include: The text sequence contained in the local feature is mapped into the query variable Q, the key variable K and the value variable V, and the specific calculation process is as follows: Q=SW Q ,K=SW K ,V=SW V , Among them, W Q , W K , W V are the linear transformation matrices of the query variable Q, key variable K and value variable V respectively, and S is the text sequence matrix contained in the local features.

6. A resume classification device based on a self-attention mechanism, used to implement the resume classification method based on a self-attention mechanism as described in any one of claims 1 to 5, characterized in that: It includes a resume information extraction module and a convolutional neural network resume classification model, wherein the resume information extraction module is used to obtain resume information, perform text extraction on the resume information to obtain plain text information, perform data cleaning on the plain text information, perform information extraction on the plain text information after data cleaning to obtain job information, and output the job information to the convolutional neural network resume classification model; the convolutional neural network resume classification model is used to classify the job information to obtain resume classification results; The convolutional neural network resume classification model includes an embedding layer, a convolutional layer, a self-attention layer, a pooling layer and a fully connected layer; the embedding layer obtains the corresponding word embedding vector according to the working information of the input convolutional neural network resume classification model, and inputs it into the convolutional layer; the convolutional layer performs a one-dimensional convolution operation on the word embedding vector to extract its local features, and inputs the local features into the self-attention layer; the self-attention layer uses the self-attention mechanism to obtain long-distance dependency information for the local features, and further captures the key classification features in the resume text according to the long-distance dependency information; the pooling layer uses the maximum pooling operation to downsample the outputs of the self-attention layer and the convolutional layer respectively, obtains two downsampled results, and sends the two downsampled results to the fully connected layer, the fully connected layer performs feature fusion on the two downsampled results and sends them to the Softmax function for classification, and obtains the final classification result of the resume; The embedding layer uses the trained continuous bag-of-words model in Word2Vec to obtain the word embedding vector of each word in the work information, uses the word embedding vector to form a matrix, and uses it as the input of the convolutional layer; the matrix formed by the word embedding vector is represented as a two-dimensional matrix X of n×d dimensions, and its expression is: X=[x1,x2,···x i ,···,x n ] T ∈R n×d , Where n is the number of word embedding vectors contained in the work information, d is the word embedding vector dimension, and x i is the word embedding vector of the i-th word in the job information; The fused feature vector is sent to the Softmax function for classification, and its calculation formula is: y i =softmax(∑W s ·V+b s ), where y i Represents the category obtained after classification prediction, W s is the weight matrix of the fully connected layer, V is the feature vector after feature fusion, b s is the bias of the fully connected layer; The self-attention layer first maps the text sequence contained in the local features into query variables Q, key-value variables K, and value variables V, then performs a dot product operation on the query variable Q and the key-value variable K, and then normalizes the dot product operation result, and uses the normalized value as the weight coefficient of the value variable. Finally, the weight coefficient and the value variable are weighted and summed, and then processed with the softmax function to obtain the output of the self-attention layer; The text sequence contained in the local feature is mapped into the query variable Q, the key variable K and the value variable V, and the specific calculation process is as follows: Q=SW Q ,K=SW K ,V=SW V , Among them, W Q , W K , W V are the linear transformation matrices of the query variable Q, key variable K and value variable V respectively, and S is the text sequence matrix contained in the local features.

Images