Open-source crowd behavior mining and measurement method based on markov model and entropy

Abstract

The application discloses an open source crowd wisdom behavior mining and measuring method based on a Markov model and entropy, uses the Markov model to model a work flow of an open source project, uses a feature probability transition matrix to cluster the model, and obtains a representative behavior model, i.e., a representative cluster model in the project; the representative cluster model in the project is used to calculate a corresponding source entropy, and a weighted source entropy of the project is obtained by weighting, the weighted source entropy is used as an indicative index for measuring the organization of the development behavior of the project, is used for guiding collaborative development of the open source project, and realizes early warning of an inefficient link. The method has the advantages of being simple, accurate and practical, and can help developers to better collaboratively develop, strengthen cooperation and communication, accelerate problem solving efficiency and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of software development, and particularly relates to an open source crowd behavior mining and measurement method based on a Markov model and entropy. BACKGROUND

[0002] Recently, with the development of network technology, the open source software development method gradually becomes popular, and more and more software selects an online cooperation form for development. The open source development cooperation form solves the problems caused by software cooperation development in different regions and at different times, greatly facilitating the development of complex software. Through the version control of open source software, the task proposal, discussion and audit mode, the software development can be more efficiently completed; the multi-person cooperation mode can also attract more innovation and increase the flexibility of software development.

[0003] However, the open source development method is not perfect. With the increase of the number of cooperators, the cost of communication increases, which will inevitably make the development process and turnover more time-consuming and laborious. How to reduce the efficiency reduction problem caused by the increase of the scale of software development has always been the concern of the open source field. The development task workflow record in the open source software is important information of the open source software. By analyzing the cooperation mode in the development task workflow record, the organization degree of open source cooperation can be evaluated, and the inefficient link can be reminded, which has great significance for open source software cooperation. SUMMARY

[0004] The technical problem to be solved by the application is to provide an open source crowd behavior mining and measurement method based on a Markov model and entropy to solve the above problems in the prior art.

[0005] To achieve the above technical purpose, the technical scheme adopted by the application is as follows:

[0006] The open source crowd behavior mining and measurement method based on a Markov model and entropy comprises:

[0007] The Markov model is used to model the workflow of the open source project, and the feature probability transition matrix is used to cluster the model to obtain a representative behavior model, i.e. a representative cluster model in the project.

[0008] The source entropy corresponding to the representative cluster model in the project is calculated, and the weighted source entropy of the project is obtained by weighting, and the weighted source entropy is used as an indicative index for measuring the organization of the development behavior of the project, which is used to guide the cooperative development of the open source project and realize the early warning of inefficient links.

[0009] To optimize the above technical scheme, the specific measures adopted are as follows:

[0010] The method comprises the following steps:

[0011] Step 1: Obtain the required work behavior process data from the open source project. Assume that data of N open source projects are obtained from the open source software repository. The set Z represents all open source projects, where Z = {z1, z2, ... z i ,…,z N}, i = 1, ..., N, where N is the number of open source projects, z i represents the i-th open source project in the obtained data set;

[0012] Step 2: Obtain workflow records of all open source projects in set Z, process all workflow data, and store them in the workflow database W;

[0013] Step 3: Get data from the workflow database W. For the jth record w of the i-th project in W, ij , after screening and integration, it is abstracted to obtain the workflow sequence s ij , use Markov chain to build the model m corresponding to each workflow record ij , the set of all workflow sequences is S, and the set of all models is M;

[0014] Step 4: For the i-th open source project, according to the obtained sub-model set M i , cluster it using the hierarchical clustering algorithm and get K clusters C i1 , C i2 ,…,C ij ,…,C ik , where the jth cluster C ij Including ij The sub-model corresponding to each workflow sequence is recalculated, and the corresponding cluster Markov chain model m'ij is obtained after all sub-models in the cluster are recalculated. This is the representative cluster model in the project, and the set of all cluster models is M'.

[0015] Step 5: Calculate the source entropy for the cluster Markov chain model in the i-th open source project. First, use the source entropy calculation formula to calculate the source entropy of each cluster model, and then perform weighted summation based on the number of workflow records contained in each cluster to obtain the source entropy H of the project calculated for K clusters. i ;

[0016] Step 6: Based on the calculated source entropy of the open source project, obtain an indication of the organization of the project development. Use the source entropy of the project to indicate the organization of the project. At the same time, find the workflow records that increase the source entropy and mark these records with warnings and special attention labels.

[0017] The step 1 above obtains the required work behavior flow data from the open source platform, the work flow data of each project obtained meets the data size scale, and the obtained information of the open source project is filtered and integrated, and finally the flow data obtained includes the time, developer and development content information of the project.

[0018] The step 2 above obtains all work behavior flow records contained in each open source project Z i in the set Z, removes irrelevant redundant information for each work behavior flow record, and only retains the specific description of the behavior, i.e. time, person and content, and then stores in the database W.

[0019] The step 3 above specifically includes the following steps:

[0020] Step 3.1: Obtain data from the work flow behavior database W, the jth record of the ith project in W is denoted as w ij , and perform sequence processing on each work flow record;

[0021] Step 3.2: For the behaviors in w ij that occur at the same time and concurrently, sort them according to fixed order rules;

[0022] Step 3.3: For the behavior types in w ij , classify different types of behaviors and mark them with different category labels;

[0023] Step 3.4: For the developers in w ij , according to the behaviors made by the developers, whether they involve the project development and interesting content, divide them into two roles of core developers and ordinary developers;

[0024] Step 3.5: Integrate the information obtained by step 3.2, step 3.3 and step 3.4 on w ij to obtain the sequence data of the work flow behavior record, i.e. the work flow sequence s ij , and the behaviors are arranged in the order of events, and each behavior contains the following two-tuple information: (r, t), wherein r is the role category of the developer of the behavior, and t is the behavior category of the behavior;

[0025] Step 3.6: Model the sequence s ij using Markov chain, and calculate the parameters contained in the Markov chain model m ij according to the content of the sequence, including state set S, initial distribution Π and probability transition matrix P.

[0026] The step 4 above specifically includes the following steps:

[0027] Step 4.1: For the ith open source project, according to the obtained sub-model set M i , initialize a cluster corresponding to each sub-model;

[0028] Step 4.2: Use a hierarchical clustering algorithm, merge two most similar clusters in each iteration until the number of clusters is K, and obtain the result as clusters C i1 , C i2 , …, C ik , the jth cluster C ij includes n ij sub-models corresponding to workflow sequences;

[0029] Step 4.3: Calculate the corresponding cluster Markov chain model m' ij and the parameters of the model for each cluster in the K clusters, including the state set S, the initial distribution Π, and the probability transition matrix P.

[0030] The above step 5 specifically includes the following steps:

[0031] Step 5.1: For the ith open source project, according to the model parameters, calculate the source entropy of the cluster Markov chain model;

[0032] Step 5.2: Calculate the source entropy of each cluster model, assuming that the parameters of the model are the state set S, the initial distribution Π, and the probability transition matrix P, then the source entropy h = Π × Σ p t ij × log p t ij , where p t is the probability transition matrix corresponding to the Markov chain model at time t, p t ij is the transition probability from state i to state j at time t;

[0033] Step 5.3: Weighted sum the source entropy of the K cluster models within the project, the weight value corresponding to each cluster model is related to the number of behavior workflow sequences contained in the model, assuming that there are M workflow sequences in the ith open source project, clusters C i1 , C i2 , …, C ik , the jth cluster C ij includes n ij workflow sequences, then the corresponding weight value is n ij / M, and the source entropy of the project is

[0034] The above step 6 specifically includes the following steps:

[0035] Step 6.1: According to the source entropy of the open source project, the organization of the open source project development behavior is analyzed and evaluated, and an indication of the organization of the project development is obtained: the higher the source entropy, the better the organization of the project, and vice versa, the lower the source entropy, the worse the organization of the project;

[0036] Step 6.2: Compare the projects with high source entropy and the projects with low source entropy, find the difference between the process models of the two types of projects in the behavior link, and mark the behavior link with warning and key attention label if the difference is higher than the set range, indicating that these links have potential adverse effects on the development of the open source project.

[0037] The present application has the following beneficial effects:

[0038] The present application uses Markov model to model the workflow of the open source project, and uses the feature probability transition matrix to cluster the model to obtain a representative behavior model, i.e. a representative cluster model in the project; the source entropy corresponding to the representative cluster model in the project is calculated, and the weighted source entropy of the project is obtained by weighting, which is used as an indicative index of the organization of the project development behavior, for guiding the collaborative development of the open source project, and realizing the early warning of inefficient links. Based on the development task workflow record in the open source software, the cooperation mode in the record is analyzed to realize the organization degree evaluation of the open source collaboration, and the inefficient links can be reminded, and the problem of efficiency reduction caused by the increase of the software development scale can be reduced. The present application has the advantages of simplicity, accuracy and practicability, and can help the developers to better collaborate and develop, strengthen the cooperation and communication, and speed up the problem solving efficiency, etc. BRIEF DESCRIPTION OF DRAWINGS

[0039] Fig. 1 The present application is a method flowchart;

[0040] Fig. 2 The present application is a logic diagram. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0042] The steps in the present application are arranged by using labels, but are not used to limit the sequence of the steps, unless the sequence of the steps is explicitly described or the execution of a step needs other steps as a basis, otherwise the relative sequence of the steps can be adjusted. It can be understood that the term "and / or" used herein involves and covers any and all possible combinations of one or more of the associated listed items.

[0043] As Figs. 1-2As shown, the open source crowd wisdom behavior mining and measuring method based on a Markov model and entropy includes a work behavior process modeling and clustering step for an open source project and a source entropy measuring step for a representative model of the open source project.

[0044] In the work behavior process modeling and clustering step for the open source project, a Markov model is used to model the work process of the open source project, and a feature probability transition matrix is used to cluster the work process model to obtain a representative behavior model, i.e., a representative clustering model in the project.

[0045] In the source entropy measuring step for the representative model of the open source project, the source entropy measuring step for the representative model of the open source project: calculates the corresponding source entropy of the representative clustering model in the project, and obtains the weighted source entropy of the project according to the size of the clustering, and uses the weighted source entropy as an indicative index for measuring the organization of the development behavior of the project, to guide the collaborative development of the open source project and realize the early warning of inefficient links.

[0046] It should be noted that the hardware environment of the present application includes a router or switch capable of normal access to a wide area network, and a computer capable of normal use. The software environment includes Python3.10.

[0047] Specifically, the open source crowd wisdom behavior mining and measuring method based on a Markov model and entropy provided by the embodiments of the present application includes the following steps:

[0048] Step 1: Obtain the required work behavior process data from the network data transmission interface provided by the open source platform (such as Github, Gitee, etc.). It is assumed that N open source project data is obtained from the open source software repository, and the entire set of all open source projects is represented by set Z, Z={z1,z2,…z i ,…,z N}, i=1,…,N, where N is the number of open source projects, and z i represents the i-th open source project in the obtained data set.

[0049] Step 2: Obtain the work flow records of all open source projects in set Z, and store all the work flow data in a certain organization form in database W (such as Oracle, MySQL, etc.) after processing;

[0050] Step 3: Obtain the issue data of the project from the work flow database W. For the j-th record w ij of the i-th project in W, filter out the issue records that do not meet the condition of being too short and information missing, merge redundant events, expand concurrent events, and obtain the work flow sequence s ij , and use a Markov chain to establish a model m corresponding to each work flow record.ij where S is the set of all workflows, and M is the set of all models;

[0051] Step 4: For the i-th open source software, according to the obtained set of sub-models M i , use the hierarchical clustering algorithm to cluster them, and obtain K clusters C i1 , C i2 , …, C ik , the j-th cluster C ij includes n ij sub-models corresponding to workflow sequences, and the corresponding cluster Markov chain model m'ij is recalculated after the comprehensive class of all sub-models in the cluster, where the set of all cluster models is M', and the size of K is usually between 5-10;

[0052] Step 5: Calculate the source entropy of the cluster Markov chain model in the i-th open source project, first calculate the source entropy of each cluster model using the source entropy calculation formula, and then weighted sum according to the number of workflow records contained in each cluster, to obtain the source entropy H i of the project calculated by K clusters;

[0053] Step 6: According to the calculated source entropy H of the open source project, different projects are divided into high-entropy projects and low-entropy projects, and the Markov chain process models of the two types of projects are compared to find the workflow records that increase the source entropy, and these records are marked with warning and focus labels.

[0054] When applying the open source crowd behavior mining and measurement method based on Markov model and entropy disclosed in the present application, that is, when applying the open source software recommendation method based on topic association relationship analysis disclosed in the present application, first, a certain number of behavior event records are needed. Then use the Markov chain model to model the workflow records of the open source project. Next, the sub-models are clustered to obtain representative process models within the project. Finally, the source entropy calculation evaluation method is used to evaluate the organization of the open source project, and the inefficient links are found out by comparing the high and low efficiency process models to give a prompt.

[0055] A typical application scenario of the present application is to analyze and evaluate some open source projects. For example: for an open source project with gradually increasing size and a large number of development product iteration requirements, there are many unprocessed events, and through the intervention of the method, the historical work behavior is analyzed to obtain an organization evaluation of the project. At the same time, a template project with efficient cooperation is given for comparison, and by referring to the efficient cooperation mode, the ongoing event processing can be correctly guided, and the development problems caused by inefficient communication and cooperation can be avoided.

[0056] In the embodiment, the step 1 obtains the required work behavior flow data from the open source platform, and the work flow data of each project must have a certain data size, and the obtained open source project information is filtered and integrated, and finally the obtained flow data includes the information of the project time, developer, development content and the like.

[0057] The step 2 obtains all work behavior flow records contained in each open source project Z i in the set Z, removes irrelevant redundant information for each work behavior flow record, and only retains the specific description of the behavior, that is, the time, person, content and the like, and then stores in the database W.

[0058] The step 3 specifically includes the following steps.

[0059] Step 3.1: obtaining data from the work flow behavior database W, the jth record of the ith project in W is denoted as w ij , and each work flow record is subjected to serialization processing.

[0060] Step 3.2: for the behaviors appearing at the same time in w ij , the behaviors are sorted according to the fixed order (such as dictionary order) rule to avoid the situation that the behaviors appearing at the same time are in disorder;

[0061] Step 3.3: for the behavior types in w ij , different types of behaviors are classified and labeled with different class labels.

[0062] Step 3.4: for the developers in w ij , according to the behaviors made by the developers, whether the behaviors involve the project development and the interesting content, the developers are divided into two roles of core developers and ordinary developers.

[0063] Step 3.5: integrating the information obtained by the step 3.2, the step 3.3 and the step 3.4 to w ij to obtain the serialized data of the work flow behavior record, that is, the work flow sequence s ij , and the behaviors are arranged according to the event sequence, and each behavior unit contains the following two-tuple information: (r, t), wherein r is the role category of the developer of the behavior, and t is the behavior category of the behavior.

[0064] Step 3.6: modeling the sequence s ij using Markov chain, and calculating the parameters contained in the Markov chain model m ij according to the content of the sequence, including the state set S, the initial distribution Π and the probability transition matrix P.

[0065] The step 4 specifically includes the following steps.

[0066] Step 4.1: For the ith open source project software, according to the obtained sub-model set M i , initialize a cluster corresponding to each sub-model;

[0067] Step 4.2: Use a hierarchical clustering algorithm, merge two most similar clusters in each iteration until the number of clusters is K, and obtain the result as clusters C i1 , C i2 , …, C ik , the jth cluster C ij includes n ij sub-models corresponding to workflow sequences;

[0068] Step 4.3: Calculate the corresponding cluster Markov chain model m' ij and the parameters of the model for each cluster of the project, including the state set S, the initial distribution Π, and the probability transition matrix P.

[0069] The step 5 specifically includes the following steps:

[0070] Step 5.1: For the cluster Markov chain model in the ith open source project, calculate the source entropy according to the model parameters obtained in step 4.3;

[0071] Step 5.2: Calculate the source entropy of each cluster model, assuming that the parameters of the model are the state set S, the initial distribution Π, and the probability transition matrix P, then its source entropy h = Π × Σ p t ij × log p t ij , where p t is the probability transition matrix corresponding to the Markov chain model at time t, p t ij is the transition probability from state i to state j at time t;

[0072] Step 5.3: Weighted sum the source entropy of the K cluster models within the project, the weight value corresponding to each cluster model is related to the number of behavior workflow sequences contained in the model, assuming that there are M workflow sequences in the ith open source project, clusters C i1 , C i2 , …, C ik , the jth cluster C ij includes n ij workflow sequences, then its corresponding weight value is n ij / M, and the source entropy of the project is

[0073] The step 6 specifically includes the following steps:

[0074] Step 6.1: According to the calculated source entropy H of the open source project, the organization of the open source project development behavior is analyzed and evaluated, and an indication of the organization of the project development is obtained: the higher the source entropy H, the better the organization of the project, and vice versa, the lower the source entropy H, the worse the organization of the project;

[0075] Step 6.2: Compare the projects with high source entropy and the projects with low source entropy, find the differences between the process models of the two types of projects in the behavior link, and mark the behavior link with warning and focus attention tags if the difference is too high, indicating that these links have potential adverse effects on the development of the open source project.

[0076] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0077] In addition, it should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be combined appropriately to form other embodiments that those skilled in the art can understand.

Claims

1. An open-source crowd intelligence behavior mining and measurement method based on Markov models and entropy, characterized by: include: Use the Markov model to model the workflow of the open source project and cluster the model using the characteristic probability transition matrix to obtain a representative behavior model, that is, a representative clustering model in the project; The corresponding information source entropy of each representative clustering model in the project is calculated and weighted to obtain the weighted information source entropy of the project. The weighted information source entropy is used as an indicative indicator to measure the organization of the project's development behavior, guiding the collaborative development of open source projects and realizing early warning of inefficient links. The method specifically comprises the following steps: Step 1: Obtain the required work behavior process data from the open source project. Assume that data of N open source projects are obtained from the open source software repository. The set Z represents all open source projects, where Z = {z1, z2, ... z i ,…,z N }, i = 1, ..., N, where N is the number of open source projects, z i represents the i-th open source project in the obtained data set; Step 2: Obtain workflow records of all open source projects in set Z, process all workflow data, and store them in the workflow database W; Step 3: Get data from the workflow database W. For the jth record w of the i-th project in W, ij , after screening and integration, it is abstracted to obtain the workflow sequence s ij , use Markov chain to build the model m corresponding to each workflow record ij , the set of all workflow sequences is S, and the set of all models is M. Specifically, it includes the following steps: Step 3.1: Get data from the workflow behavior database W. The jth record of the i-th project in W is denoted as w ij , serialize each workflow record; Step 3.2: For w ij The behaviors that occur concurrently at the same time are sorted according to fixed order rules; Step 3.3: For w ij The behavior types in the ,classify different types of behaviors and label them with different categories; Step 3.4: For w ij Developers in the project are divided into two roles: core developers and ordinary developers, based on their actions and whether they involve project development and interesting content; Step 3.5: Compare steps 3.2, 3.3, and 3.4 to w ij The processed information is integrated to obtain the serialized data of the workflow behavior record, that is, the workflow sequence s ij , and the behaviors are arranged in the order of events. Each behavior contains the following two-tuple information: (r, t), where r is the developer role category of the behavior and t is the behavior category of the behavior; Step 3.6: Use Markov chain to train the sequence s ij Perform modeling and calculate the Markov chain model m based on the content of the sequence ij The parameters included include the state set S, the initial distribution Π, and the probability transition matrix P; Step 4: For the i-th open source project, according to the obtained sub-model set M i , cluster it using the hierarchical clustering algorithm and get K clusters C i1 , C i2 ,…,C ij ,…,C ik , where the jth cluster C ij Including ij The sub-model corresponding to each workflow sequence is recalculated, and the corresponding cluster Markov chain model m'ij is obtained after all sub-models in the cluster are recalculated. This is the representative cluster model in the project, and the set of all cluster models is M'. Step 5: Calculate the source entropy for the cluster Markov chain model in the i-th open source project. First, use the source entropy calculation formula to calculate the source entropy of each cluster model, and then perform weighted summation based on the number of workflow records contained in each cluster to obtain the source entropy H of the project calculated for K clusters. i , specifically including the following steps: Step 5.1: For the clustering Markov chain model in the i-th open source project, calculate the source entropy based on the model parameters; Step 5.2: Calculate the source entropy of each clustering model. Assuming that the parameters of the model are the state set S, the initial distribution Π, and the probability transfer matrix P, then its source entropy h=Π×Σp t ij ×log p t ij , where p t is the probability transfer matrix corresponding to the Markov chain model at time t, p t ij is the transition probability from state i to state j at time t; Step 5.3: Perform weighted summation of the source entropy of the K clustering models within the project. The weight corresponding to each clustering model is related to the number of behavioral workflow sequences contained in the model. Assume that there are a total of M workflow sequences within the i-th open source project, and cluster C i1 , C i2 ,…,C ik , the jth cluster C ij Including ij workflow sequence, then its corresponding weight is n ij / M, the source entropy of the project is Step 6: Based on the calculated source entropy of the open source project, obtain an indication of the organization of the project development. Use the source entropy of the project to indicate the organization of the project. At the same time, find the workflow records that increase the source entropy and mark these records with warnings and special attention labels.

2. The open source crowd intelligence behavior mining and measurement method based on Markov model and entropy according to claim 1 is characterized in that: The step 1 obtains the required work behavior process data from the open source platform, obtains the workflow data of each project to meet the data volume scale, and filters and integrates the information of the obtained open source projects. The final process data includes the project time, developer, and development content information.

3. The open source crowd intelligence behavior mining and measurement method based on Markov model and entropy according to claim 1 is characterized in that: Step 2 is performed on each open source project Z in the set Z. i , obtain all the work behavior process records it contains, and for each work behavior process record, remove irrelevant redundant information and only retain the specific description of the behavior, namely the time, person, and content, and then store it in the database W.

4. The open source crowd intelligence behavior mining and measurement method based on Markov model and entropy according to claim 1 is characterized in that: The step 4 specifically includes the following steps: Step 4.1: For the i-th open source project, according to the obtained sub-model set M i , initialized to correspond to one cluster for each sub-model; Step 4.2: Use the hierarchical clustering algorithm to merge the two most similar clusters in each iteration until the number of clusters is K, and the result is cluster C. i1 , C i2 ,…,C ik , the jth cluster C ij Including ij A sub-model corresponding to each workflow sequence; Step 4.3: Calculate all sub-models in each cluster of K clusters to obtain the corresponding cluster Markov chain model m' ij And the parameters of the model, including the state set S, the initial distribution Π, and the probability transfer matrix P.

5. The open source crowd intelligence behavior mining and measurement method based on Markov model and entropy according to claim 1 is characterized in that: The step 6 specifically includes the following steps: Step 6.1: Analyze and evaluate the organization of open source project development behavior based on the source entropy of the open source project to obtain an indication of the organization of the project development: the higher the source entropy, the better the project organization; conversely, the lower the source entropy, the worse the project organization; Step 6.2: Compare projects with high source entropy and projects with low source entropy to find out the differences in the behavioral links of the process models between the two types of projects. Label the behavioral links where the differences are higher than the set range with warnings and special attention labels, indicating that these links have the potential to adversely affect the development of the open source project.

Images