A deep learning-based software crowdsourcing task recommendation method

Abstract

The application discloses a software crowdsourcing task recommendation method based on deep learning. At present, many related researches propose to use the deep learning method to recommend the crowdsourcing task text information, but in the existing method, the extraction method of the crowdsourcing task text information lacks universality, and due to the unbalanced distribution characteristics of the crowdsourcing data, the hit rate and diversity cannot be considered in the index of the recommendation result. The method contains three parts: extracting crowdsourcing text features based on a pre-training model Bert, further learning the crowdsourcing text features based on CNN+LSTM, and the output based on the above two models, which can adaptively overcome the loss function of the unbalanced distribution of the crowdsourcing data. Through the application, the developer recommendation of a specific software crowdsourcing platform can be realized simply and efficiently, and the hit rate and diversity of the recommendation result are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of crowdsourcing, and relates to an adaptive recommendation method for multiple types of software crowdsourcing tasks. BACKGROUND

[0002] In recent years, with the increase of software complexity, it is a common method to divide a large software development task into several small tasks. The separation steps can include: (1) dividing a large project into a series of small modules, which have simple requirements, clear interfaces, clear objectives and comprehensive testing; (2) combining all these modules into the final artifacts. This trend has laid a solid foundation for the emergence of crowdsourcing software development.

[0003] Compared with traditional outsourcing, crowdsourcing emphasizes outsourcing to unspecified volunteers in a free and voluntary form, who are assigned to a small part of a specific project. In recent years, the market has flourished, and many crowdsourcing platforms have emerged, such as Code City, Programmer Inn in China, and TopCoder abroad.

[0004] In a typical software crowdsourcing process, first, the requester publishes the software development task on the platform. Any developer can register as a "registrant" for this specific development task according to his personal interest and proficiency. This process can last for a period of time. Then, if the required requirements, specifications, etc. are clear, the "registrant" can start working after being confirmed by the requester or the platform. Finally, before the deadline, the developer should upload his final work to the platform after completing his development work, and then wait for the reviewer or the requester to determine to adopt the work of one or several registrants according to several evaluation indexes.

[0005] However, the requester has to waste a lot of time waiting for enough developers to register and choosing the right developers from them. Even so, the selected developers may not be the most suitable developers for the specific task, because "suitable developers" may miss this task due to a large number of available tasks. When a large number of developers register for the published task, the efficiency of the registrant filtering will be worse, and the quality of the final task work will be more unstable. Therefore, it is very important to recommend target developers for tasks in software crowdsourcing platforms. However, considering the particularity of software crowdsourcing platforms, traditional content-based recommendation cannot achieve good results for many reasons: first, the software crowdsourcing historical task dataset satisfies the 80-20 rule and is extremely unbalanced. Second, the number of potential developers for software crowdsourcing is too large, and traditional methods cannot effectively learn. The above reasons will make "active" developers more likely to be recommended, while relatively "inactive" developers are likely to be "buried", which obviously violates the original intention of crowdsourcing.

[0006] Many recommendation scheme designs have been proposed in the current study, which mainly extracts relevant text features for appropriate recommendation. It is usually divided into text features based on crowdsourcing tasks and mixed text features based on crowdsourcing tasks and developers; the text features based on crowdsourcing tasks often depend on some keywords or are optimized for a certain task type, and lack a certain universality; the mixed text features based on task text and developer add information about developers, but most software crowdsourcing platforms currently provide no further feedback content for requesters except for satisfaction ratings after task completion, and can only use the text information defined by the developer for modeling, but such information is not always reliable. At the same time, most methods often only consider a single developer in the model training process, while in actual application, a software crowdsourcing task often needs to be completed by multiple developers in collaboration or there are multiple candidate developers, and using only a single developer for training cannot achieve good diversity of the recommended results. The purpose of considering diversity is to encourage more developers to migrate to the task, because the essence of crowdsourcing is to utilize "collective wisdom". If the hit rate is focused on, the recommendation system is accurate in statistics, but if its diversity is poor, it may not be very useful in actual application. SUMMARY

[0007] The purpose of the present application is to further improve the recommendation method of software crowdsourcing in the ability of recommendation hit rate and diversity, and has a certain universality, and a software crowdsourcing task recommendation method based on deep learning is proposed.

[0008] On the basis of deep learning theory, the text of the crowdsourcing task is first segmented, the Bert pre-training model is used to provide word vectors for the downstream task, a two-dimensional feature is created for each crowdsourcing task, and the deep learning model composed of CNN+LSTM is input, the output result is the confidence score of each developer (label) of the corresponding platform corresponding to the crowdsourcing task, and then it is given to the loss function which can adaptively balance the label sample distribution for training. This method can overcome the imbalance of crowdsourcing task historical data, and can achieve good recommendation hit rate and diversity.

[0009] The steps of the method of the present application are:

[0010] Step 1. For a specific crowdsourcing platform, collect the text information (category, description, etc.) of the crowdsourcing task and the corresponding completed developer id (one or more). For developers, filter out developers with less than q completed tasks.

[0011] Step 2. Divide the collected and filtered data into training set and test set, the principle of division is to center on the developer (label), and preferentially ensure that each developer (label) has corresponding samples in the training set.

[0012] Step 3. The text information of the crowdsourcing task is segmented by using tools such as nltk. Since the lengths of the segmented text information of different tasks are different, the input of the model needs to be consistent in length, and a maximum length sequence_length needs to be obtained from the training set. The samples in the training set with a segmented length less than sequence_length are padded, and the samples in the test set with a segmented length greater than sequence_length are truncated.

[0013] Step 4. The segmented and padded training set samples are first input into the Bert pre-training model to generate the word vectors required by the downstream task. The generated word vectors are further input into the deep learning model of CNN+LSTM for further feature learning, and the final output is the confidence score of each developer of the crowdsourcing platform.

[0014] Step 5. The confidence score of each developer and the true result corresponding to the task are input into the loss function to calculate the loss and back-propagate the gradient.

[0015] Step 6. Repeat steps 4 to 5 until the model converges.

[0016] Advantages of the present application: The present application only needs to collect certain crowdsourcing historical task data of the crowdsourcing platform to be applied and perform appropriate preprocessing to stably realize its performance. The loss function of the present application can effectively adaptively balance the phenomenon of unbalanced distribution of developer (label) samples in the crowdsourcing data set, and implement multi-developer (label) training during the training process, which can effectively improve the hit rate and diversity of recommendation. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is the architecture diagram of the model CNN+LSTM.

[0018] Figure 2 is the internal architecture diagram of LSTM. DETAILED DESCRIPTION

[0019] The present application is further described below in conjunction with the drawings, and the specific implementation steps are as follows.

[0020] Step 1. For a specific crowdsourcing platform, collect the text information (category, description, etc.) of the crowdsourcing task and the corresponding completed developer id (one or more). For developers, filter out developers with less than q completed tasks.

[0021] The filtering out of developers with a task completion number less than q aims to ensure the quality of the recommended developers, and the screening according to the task completion number is a relatively simple and reliable way. The size of q is set according to specific circumstances. For the collected crowdsourcing task text information, due to the inconsistent formatting description of tasks on different platforms, it is necessary to splice it into a whole for unified processing

[0022] Step 2. The collected and filtered data is divided into training set and test set, and the principle of division is to take the developer (label) as the center and preferentially ensure that each developer (label) has corresponding samples in the training set.

[0023] Step 3. When the text information of the crowdsourcing task is segmented by using tools such as nltk, due to the different lengths of the segmented text information of different tasks, the input of the model needs to be consistent in length, and a maximum length sequence_length needs to be obtained from the training set. The samples in the training set with a segmentation length less than sequence_length will be padded, and the samples in the test set with a segmentation length greater than sequence_length will be truncated.

[0024] Among them, the results of segmentation can exclude some commonly used words and punctuation marks (such as "of", "we" in Chinese; "am", "is", "are" in English) according to the length of segmentation and regular expressions to further improve data quality.

[0025] Step 4. The training set samples after segmentation and padding are first input into the Bert pre-training model to generate word vectors required by the downstream task. The word vectors can well quantify the similarity between the corresponding crowdsourcing task content and the tasks. The generated word vectors are further given to the deep learning model of CNN+LSTM for further feature learning, and the final output result is the confidence score of each developer of the crowdsourcing platform. The specific introduction of the prediction model is as follows:

[0026] Step 4-1. Preliminary feature extraction of convolutional network.

[0027] As shown in Figure 1 , the length of each crowdsourcing task text forms a two-dimensional matrix x∈R L×V , where L is the text length, and V is the word vector dimension. The output of the CNN network is subjected to convolution operation, and W∈R K×V is the convolution kernel for convolution operation, where K is the n-gram size, and the feature at different positions is extracted by sliding on the sentence. A convolution kernel and a word window x i:i+K-1 are subjected to convolution operation (with a stride of 1) to obtain a series of feature fields m∈R L-K+1 , where the calculation method of each element m i is:

[0028] m i = f(x i:i+k-1 * W + b)

[0029] where * represents element-wise operation, b represents bias unit and f represents nonlinear activation function (such as ReLU, etc.), and the feature field is subjected to a dynamic maximum pooling layer to select the most prominent multiple feature values m max . This process extracts multiple features from a convolution kernel, which can use multiple convolution kernels of different sizes (different K values) to combine multiple output channels to form a high-level feature representation M ∈ R reduced_size×embed_size of the crowdsourcing task text information. Wherein, reduced_size is the length of the text sequence after convolution network processing, and embed_size is the embedding dimension of each element of the sequence.

[0030] Step 4-2. Feature extraction of LSTM recurrent neural network.

[0031] The recurrent neural network (RNN) can propagate historical information through a chain neural network structure. When processing sequence data, it will comprehensively process according to the current time step x t and the hidden state h t-1 containing the information of the previous historical time step. However, as the gap between two time steps becomes larger, the standard RNN cannot handle long dependencies, and there is a risk of gradient explosion or weakening. The LSTM network is born to alleviate these two defects. Although there are many variants of LSTM, the standard LSTM structure is still used in this application.

[0032] As shown in Figure 2 , the standard LSTM structure is like RNN, and each time step has a series of repeated modules. The input of the module is the current time step x t , the hidden state h t-1 of the previous step and the memory cell c t-1 , and the output of the module is determined by a series of gates, which are: forget gate f t , input gate i t and output gate O t . These gates jointly determine how to update the hidden state h t and the memory cell c t of the current time step and the output x′ t+1 . This application defines d as the dimension of the LSTM from input to output, and the specific calculation function of LSTM is as follows:

[0033] i t = σ(W i *[h t-1 , x t]+b i )

[0034] f t = σ(W f *[h t-1 , x t ]+b f )

[0035] q t = tanh(W q *[h t-1 , x t ]+b q )

[0036] o t = σ(W o *[h t-1 , x t ]+b o )

[0037] c t = f t ⊙c t-1 +i t ⊙q t

[0038] h t = o t ⊙tanh(c t )

[0039] where σ represents sigmoid function, which can compress the input value to [0, 1], tanh represents hyperbolic tangent function, which can compress the input to [-1, 1], and ⊙ represents element-wise operation. The following will combine the high-level feature representation M of the text information of the output of the CNN reduced_size×embed_size Further explain the mechanism behind LSTM. First, reduced_size can be regarded as the time step, and embed_size can be regarded as the dimension of the input feature. Second, f t can be regarded as having a function of controlling to what extent the historical information of the previous time step is discarded, i t can be regarded as having a function of controlling how much information of the current time step is saved to the candidate memory cell c t of the current time step, and o t can be regarded as having the ability to control the output according to the memory cell c t of the current time step. The present application utilizes the fact that LSTM can learn the comprehensive features of the front and back information of the crowdsourcing task text, takes the state information of the last time step of the LSTM output as the representation of the crowdsourcing task text information, and then connects a full connection layer to output the confidence score of the platform developer (label).

[0040] Step 5. Each developer confidence score is input into the loss function along with the true result corresponding to its task to calculate the loss and backpropagate the gradient.

[0041] The loss function of the present application does not use the conventional cross-entropy loss function for single-label classification because the present application considers that the recommendation of the crowdsourcing task is a multi-label (developer) problem, but at the same time, the present application does not use the conventional binary cross-entropy loss function for multi-label classification because the number of samples contained by different developers (labels) in the crowdsourcing data set is unbalanced, which easily leads to poor learning effect of the model. Although the weight between unbalanced labels can be manually adjusted to solve this situation, in crowdsourcing recommendation, the number of developers (labels) is large and the weight adjustment is complicated in the actual production environment, which has a large time cost. The present application uses the generalization of "softmax + cross-entropy" to multi-label classification to achieve a loss function that does not need to specially adjust the label weight and threshold, and the specific formula derivation is as follows:

[0042] First, only looking at the formula of "softmax + cross-entropy" for single-label classification:

[0043]

[0044] Where n represents the number of labels, and the confidence score of each label is s 1 ,s 2 ,……,s n , and t is the target label of the sample. The optimization direction of this loss function is to make s t become the maximum confidence score in s 1 ,s 2 ,……,s n . According to the expression on the right side of the equality, the characteristic of this loss function is that all the confidence scores s 1 ,s 2 ,……,s n of the non-target labels are compared with the confidence score of the target label, and the maximum value of the difference is as small as possible less than 0, which indirectly realizes the effect that the confidence score of the target label is greater than the confidence score of each non-target label.

[0045] Therefore, if it is generalized to the multi-label classification scene, by analogy, we can get:

[0046]

[0047] For the multi-label classification problem, especially in the crowdsourcing data, the number of developers (labels) of each sample is not fixed. Therefore, a threshold needs to be set to determine which labels to output. For this purpose, an additional 0 label is set, and it is hoped that the confidence scores of the target labels are all greater than s 0 , and the confidence scores of the non-target labels are all less than s 0 , and by analogy, we can get:

[0048]

[0049] The present application specifies s 0 That is, the threshold value is 0, and the formula is simplified as:

[0050]

[0051] This is the final form of the loss function used in the present application, which does not need to consider the label distribution imbalance phenomenon, because it is not like the conventional binary cross entropy to change the multi-label classification into multiple binary classification problems, but to compare the target label confidence score with the non-target label confidence score two by two, and with the help of the property of logsumexp, the weight of each item is automatically balanced.

[0052] Step 6. Repeat steps 4 to 5 until the model converges.

[0053] The above is the preferred implementation process of the present application, any changes made according to the present application technology, as long as the function generated does not exceed the scope of the technical scheme of the present application, belongs to the protection scope of the present application.

Claims

1. A deep learning-based software crowdsourcing task recommendation method, characterized by The method comprises the following steps: Step 1. For the crowdsourcing platform, collect the text information of the crowdsourcing task and the corresponding completed developer id, and filter; Step 2. Divide the training set and the test set for the collected and filtered data; Step 3. Tokenize the text information of the crowdsourcing task, and obtain a maximum length sequence_length from the training set; The samples with tokenization length less than sequence_length in the training set are padded, and the samples with tokenization length greater than sequence_length in the test set are truncated; Step 4. The training set samples after tokenization and padding are first input into the Bert pre-training model to generate the word vector required by the downstream task; The generated word vector is further given to the deep learning model of CNN+LSTM for further feature learning, and the finally output result is the confidence score of each developer of the crowdsourcing platform; Specifically: CNN preliminary feature extraction: input the word vector matrix of the crowdsourcing task text into CNN, extract local n-gram features through different size convolution kernels, and after ReLU activation and dynamic maximum pooling, concatenate the multi-channel output to form the high-level feature representation of the text; LSTM final feature extraction: input the high-level features output by CNN into LSTM, use its gating mechanism to capture the long-range dependence relationship of the text, integrate the context semantics, and finally take the hidden state of the last time step of LSTM as the global feature to generate the confidence score of each developer through the full connection layer; Step 5. Input each developer's confidence score and the true result corresponding to the task into the loss function to calculate the loss and back-propagate the gradient; The loss function is expressed as: wherein, denotes the i-th label confidence score, denotes the j-th label confidence score, respectively; Step 6. Repeat steps 4 to 5 until the model converges.

2. The software crowdsourcing task recommendation method based on deep learning according to claim 1, characterized in that: In step 1, for the developer, filter out the developers whose completed task quantity is less than the set value.

3. The software crowdsourcing task recommendation method based on deep learning according to claim 1, characterized in that: In step 2, the principle of division is to center on the developer, and to preferentially ensure that each developer has corresponding samples in the training set.

4. The software crowdsourcing task recommendation method based on deep learning according to claim 1, characterized in that: In step 3, the text information of the crowdsourcing task is tokenized by using the nltk tool.

Images