Enterprise label clustering method combined with semantic vector fine tuning

Abstract

The invention relates to the technical field of natural language processing, in particular to an enterprise label clustering method combined with semantic vector fine tuning. Obtaining enterprise operation scope data, preprocessing the data, and calculating a general semantic feature vector; setting a fine tuning optimization target to perform fine tuning on the feature vector, and constraining the fine tuning process; segmenting the data; estimating an initial value of the DBSCAN by using a clustering result of kmeans, and performing clustering to obtain a clustering result; by means of the mode, extraction optimization of the semantic feature vectors and automatic optimization of the clustering parameters are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an enterprise label clustering method combined with semantic vector fine-tuning. Background Art

[0002] The industrial and commercial information disclosed by enterprises usually includes information such as the business scope of the enterprise. However, such information has problems such as non-standard format and inconsistent wording, which makes it difficult to directly analyze and use such information. Therefore, it is necessary to pre-clean such information.

[0003] Currently, cluster analysis can group words with the same meaning together, so it is a better way to handle the business scope (business) labels of enterprises. In the process of text clustering, the extraction of semantic feature vectors (word embedding) is particularly important. Although various general embeddings that have emerged with large models have very good performance, their performance in vertical fields is still difficult to meet commercial standards. Another difficulty in clustering is the complexity of its algorithm. Usually, when facing large-scale data, the data will be split and clustered multiple times, but the clustering parameters under different corpora are difficult to define.

[0004] In summary, it is necessary to propose an enterprise label clustering method that realizes the extraction optimization of semantic feature vectors and automatic optimization of clustering parameters combined with semantic vector fine-tuning. Summary of the invention

[0005] The purpose of the present invention is to provide an enterprise label clustering method combined with semantic vector fine-tuning to achieve extraction optimization of semantic feature vectors and automatic optimization of clustering parameters.

[0006] To achieve the above object, the present invention adopts an enterprise label clustering method combined with semantic vector fine-tuning, comprising the following steps:

[0007] Obtain enterprise business scope data, pre-process the data, and calculate the general semantic feature vector;

[0008] Set up fine-tuning optimization objectives to fine-tune the feature vector and constrain the fine-tuning process;

[0009] Split the data;

[0010] The clustering results of kmeans are used to estimate the initial value of DBSCAN, and clustering is performed to obtain the clustering results.

[0011] Among them, in the step of obtaining the enterprise business scope data, preprocessing the data, and calculating the general semantic feature vector:

[0012] Obtain the enterprise business scope data, perform word segmentation on the enterprise business scope data, and filter the data;

[0013] Count the word frequency information of tags;

[0014] Calculate the universal semantic feature vector of each business word;

[0015] Unify the format of common semantic feature vectors.

[0016] Among them, in the step of setting the fine-tuning optimization target to fine-tune the feature vector and constraining the fine-tuning process:

[0017] Set up fine-tuning optimization objectives. The formula for fine-tuning optimization objectives is:

[0018]

[0019] Where D is the positive sample space, and D' is the negative sample space;

[0020] For the positive sample space, W is the current business word, and c is the words around the current business word;

[0021] For the negative sample space, W is the current business word, and c is a business word with a different meaning from W.

[0022] Among them, in the step of setting the fine-tuning optimization target to fine-tune the feature vector and constraining the fine-tuning process:

[0023] The fine-tuning is constrained, and the constraint formula is:

[0024]

[0025] Where w, w' are the original vector and the adjusted vector respectively; λ is a hyperparameter.

[0026] Among them, after setting the fine-tuning optimization target to fine-tune the feature vector and constraining the fine-tuning process:

[0027] The optimization goal is:

[0028] L1+L2.

[0029] Among them, in the step of estimating the initial value of DBSCAN using the clustering result of kmeans and performing clustering to obtain the clustering result:

[0030] Roughly estimate the initial value of kmeans based on the data scale and perform clustering;

[0031] After K-means clustering is completed, the maximum distance from each point in each cluster to the cluster center is counted and arranged in order from low to high, the inflection point information is obtained, the cluster before the inflection point is queried, and data calculation is performed on the queried cluster; the data calculation includes the average number of clusters, the maximum radius of the cluster, the minimum radius of the cluster, the maximum distance of each point in the cluster, and the minimum distance of each point in the cluster;

[0032] The average number of clusters is used as the MinPts parameter of DBSCAN, and the maximum radius and minimum radius of the cluster are used as the parameter search interval of DBSCAN. Within this interval, with a step size of 0.01, the clustering result with the largest number of clusters is found.

[0033] Among them, after the step of finding the clustering result with the largest number of clusters:

[0034] Adjust the clustering results: select a representative word and make it the cluster center; the cluster center is determined by the frequency of the business word and the position of the word in the cluster. The determination rule is:

[0035] sigmod(freq)×dist;

[0036] Where freq is the frequency of the business word, and dist is the distance from the business word to the geometric cluster center;

[0037] Preset the threshold t, iterate each data point, calculate the distance d between the data and the cluster center, compare the distance d and the threshold t, and remove the data;

[0038] If the distance d>threshold t, the data is removed from the cluster.

[0039] Among them, after using the clustering result of kmeans to estimate the initial value of DBSCAN and perform clustering to obtain the clustering result:

[0040] Perform secondary clustering.

[0041] Among them, in the step of performing secondary clustering:

[0042] After all data are clustered, data selection is performed on clustered data, non-clustered data, and data with less than 4 clusters, and secondary clustering is performed after data segmentation.

[0043] After all data are clustered, data selection is performed for clustered data, non-clustered data, and data with less than 4 clusters. In the step of performing secondary clustering after data segmentation:

[0044] For clustered data, select the three data closest to the cluster center;

[0045] For data that is not clustered and has a cluster number less than 4, select all such data.

[0046] The enterprise label clustering method combined with semantic vector fine-tuning of the present invention obtains enterprise business scope data, pre-processes the data, and calculates a general semantic feature vector; sets a fine-tuning optimization target to fine-tune the feature vector and constrains the fine-tuning process; segments the data; uses the clustering result of kmeans to estimate the initial value of DBSCAN, and performs clustering to obtain a clustering result; through the above method, the extraction optimization of the semantic feature vector and the automatic optimization of the clustering parameters are realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flowchart of the steps of the enterprise tag clustering method combined with semantic vector fine-tuning of the present invention.

[0049] Figure 2 It is a step flow chart of S100 of the present invention.

[0050] Figure 3 It is a step flow chart of S400 of the present invention.

[0051] Figure 4 It is a schematic diagram of the K-means clustering result analysis of the present invention. DETAILED DESCRIPTION

[0052] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0053] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0054] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0055] See also Figure 1 to Figure 4 ,in Figure 1 This is a flowchart of the steps of the enterprise label clustering method combined with semantic vector fine-tuning. Figure 2 This is the step flow chart of S100. Figure 3 This is the step flow chart of S400. Figure 4 It is a schematic diagram of K-means clustering result analysis.

[0056] The present invention provides an enterprise tag clustering method combined with semantic vector fine-tuning, comprising the following steps:

[0057] S100: Obtain the business scope data of the enterprise, pre-process the data, and calculate the general semantic feature vector;

[0058] S200: Setting a fine-tuning optimization target to fine-tune the feature vector and constraining the fine-tuning process;

[0059] S300: segmenting the data;

[0060] S400: using the clustering result of kmeans to estimate the initial value of DBSCAN, and performing clustering to obtain the clustering result;

[0061] S500: performing secondary clustering.

[0062] In this embodiment, the business scope data of the enterprise is first obtained, and the data is preprocessed to calculate the general semantic feature vector. Then, a fine-tuning optimization target is set to fine-tune the feature vector and constrain the fine-tuning process. Then the data is segmented. The clustering result of kmeans is used to estimate the initial value of DBSCAN, and clustering is performed to obtain the clustering result. Finally, secondary clustering is performed. In the above manner, the extraction optimization of the semantic feature vector and the automatic optimization of the clustering parameters are achieved.

[0063] In the steps of obtaining the enterprise business scope data, preprocessing the data, and calculating the general semantic feature vector:

[0064] S101: Acquire the business scope data of the enterprise, perform word segmentation processing on the business scope data of the enterprise, and filter the data;

[0065] S102: Count the word frequency information of the tag;

[0066] S103: Calculate the general semantic feature vector of each business word;

[0067] S104: Unify the format of the common semantic feature vector.

[0068] In this embodiment, the business scope data of the enterprise is first obtained, the business scope data of the enterprise is segmented, and the data is filtered. For example, the content in the brackets of "food sales (only pre-packaged food is sold)" can be filtered. Then the word frequency information of the label is counted to assist in selecting the cluster center word. Then the common semantic feature vector of each business word is calculated through the open source embedding algorithm. Finally, the format of the common semantic feature vector is unified, such as the format of {"Company ID":[{"Label ID":{"Label Name":"","Label Vector":2.3254109842...,"Word Frequency":30,"Industry":"Industry Name"}},…]}, for subsequent processing.

[0069] Furthermore, in the step of setting a fine-tuning optimization target to fine-tune the feature vector and constraining the fine-tuning process:

[0070] Set up fine-tuning optimization objectives. The formula for fine-tuning optimization objectives is:

[0071]

[0072] Where D is the positive sample space, and D' is the negative sample space;

[0073] For the positive sample space, W is the current business word, and c is the words around the current business word;

[0074] For the negative sample space, W is the current business word, and c is a business word with a different meaning from W;

[0075] The fine-tuning is constrained, and the constraint formula is:

[0076]

[0077] Where w, w' are the original vector and the adjusted vector respectively; λ is a hyperparameter;

[0078] The optimization goal is:

[0079] L1+L2.

[0080] In this embodiment, since the general embedding vector cannot take into account the granularity of different business scenarios. Therefore, it is necessary to fine-tune the feature vector. The principle of fine-tuning is based on the following business facts: 1. The business labels under the same company show high correlation. 2. Companies with very different main businesses usually have no correlation between their business labels. After understanding this, for feature 1, the label co-occurrence can be used to fine-tune the feature vector, that is, the labels that appear under the same company are co-occurrence labels, and these labels are highly correlated. It should be pointed out that the concept of co-occurrence is not limited to establishing the correlation of labels under the same company, it can also express correlation across companies. For example, "elevator motor" will appear in the same company as "electric vehicle click", so we can determine that these two labels have a strong correlation. At the same time, "elevator motor" does not appear directly in the same company as "wiper motor", but "elevator motor" and "wiper motor" both co-occur with "AC motor" at a high frequency. Through the indirect relationship of "AC motor", "elevator motor" and "wiper motor" indirectly establish correlation. The fine-tuning method for this feature is to make the feature vectors of related business words close to each other. For feature 2, we can make the following inference: if two particularly similar business labels originally belong to different companies, and their main businesses are very different, then the feature vectors need to be adjusted to make them farther away from each other.

[0081] The specific solution is as follows: If after processing the business labels, each company has a list of business labels, and the elements in this list are related to each other. If the list is regarded as a sentence and the elements in the list are regarded as words, then this type of data can be processed using the language model. At this time, the fine-tuning optimization target formula is:

[0082]

[0083] Where D is the positive sample space, and D' is the negative sample space;

[0084] For the positive sample space, W is the current business word, and c is the words around the current business word;

[0085] For the negative sample space, W is the current business word, and c is a business word with a different meaning from W;

[0086] The above optimization goal means that the distance between positive samples is as close as possible, and the distance between negative samples is as far as possible. But at the same time, in order to avoid overfitting that causes the fine-tuned vector to deviate from the original semantics, it is necessary to constrain the fine-tuning. The constraint formula is:

[0087]

[0088] Where w and w' are the original vector and the adjusted vector respectively; λ is a hyperparameter, which means that the offset distance of the adjusted vector cannot be greater than a fixed value;

[0089] In summary, the final optimization goal is:

[0090] L1+L2.

[0091] After fine-tuning, the word vector can meet the needs of the current business scenario.

[0092] Furthermore, in the step of estimating the initial value of DBSCAN using the clustering result of kmeans and performing clustering to obtain the clustering result:

[0093] S401: Roughly estimate the initial value of kmeans according to the data scale and perform clustering;

[0094] S402: After the K-means clustering is completed, the maximum distance from the points in each cluster to the cluster center is counted and arranged in order from low to high, the inflection point information is obtained, the cluster before the inflection point is queried, and data calculation is performed on the queried cluster; the data calculation includes the average number of clusters, the maximum radius of the cluster, the minimum radius of the cluster, the maximum distance of each point in the cluster, and the minimum distance of each point in the cluster;

[0095] S403: The average number of clusters is used as the MinPts parameter of DBSCAN, and the maximum radius and the minimum radius of the cluster are used as the parameter search interval of DBSCAN. Within this interval, 0.01 is used as the step size to find the clustering result with the largest number of clusters;

[0096] S404: Adjust the clustering results: select a representative word and use the word as the cluster center; the cluster center is determined by the frequency of the business word and the position of the word in the cluster, and the determination rule is:

[0097] sigmod(freq)×dist;

[0098] Where freq is the frequency of the business word, and dist is the distance from the business word to the geometric cluster center;

[0099] S405: Preset a threshold t, iterate each data point, calculate the distance d between the data and the cluster center, compare the distance d with the threshold t, and remove data.

[0100] In this embodiment, the amount of data in each piece of data after segmentation varies and the scenarios are different. Therefore, for clustering, it is difficult to anchor how many clusters can be clustered, and it is difficult to use the K-means algorithm with the number of clusters as a clustering parameter. At the same time, if algorithms such as hierarchical clustering and density clustering that use distance as a hyperparameter are used, the appropriate hyperparameters are even more difficult to determine. Although parameter search can be performed through some hyperparameter search algorithms, if the initial value is not set properly, considering the data scale, this process will be very time-consuming. Therefore, the invention combines the characteristics of the K-means algorithm and the DBSCAN clustering algorithm to find the appropriate Eps (domain radius) and MinPts (minimum number of clusters). The specific process is as follows:

[0101] Perform K-means clustering on the data, and use the kmeans clustering results to estimate the initial value of DBSCAN: First perform K-means clustering on the data. After experimental observation, it is more reasonable to set the K value to log(N)*10, where N is the number of data points to be clustered. It should be noted that the purpose of this clustering is only to find a reasonable clustering parameter in DBSCAN, and it is only necessary to ensure that K is smaller than the optimal number of clusters, so it is reasonable to estimate a slightly smaller number.

[0102] After K-means clustering is completed, the maximum distance from each point in each cluster to the cluster center is counted and arranged in order from low to high to obtain the inflection point information. Since the number of clusters is less than the optimal number of clusters, this means that in the clustering process, some business words will inevitably be classified into other clusters because there are no more clusters to be classified. According to the clustering principle of kmeans, once "abnormal points" are added, the semantic space of the cluster will become larger, which will attract more data points to join, resulting in the content of this cluster being messy. At the same time, a large number of "abnormal points" are added, which means that the distance from the points in this cluster to the cluster center will become larger. After clustering is completed, the maximum distance from each point in each cluster to the cluster center is counted and arranged in order from low to high. This value will reach an obvious inflection point at a certain point. This inflection point usually indicates this messy cluster, and the cluster before the inflection point can be considered as a relatively reasonable cluster. The clusters before the inflection point are queried, and data calculation is performed on the queried clusters; the data calculation includes the average number of clusters, the maximum radius of the cluster, the minimum radius of the cluster, the maximum distance between the points in the cluster, and the minimum distance between the points in the cluster.

[0103] like Figure 4 As shown in the figure, kmeans clustering result analysis. The ordinate is the maximum distance from each point to the cluster center in different clusters, and the abscissa is the index of the sorted cluster. It can be found that the image has an inflection point around index 150, so the cluster before the inflection point is the object of parameter analysis.

[0104] The average number of clusters is used as the MinPts parameter of DBSCAN, and the maximum radius and minimum radius of the cluster are used as the parameter search interval of DBSCAN. Within this interval, with a step size of 0.01, the clustering result with the largest number of clusters is found.

[0105] In the clustering results, sometimes the number of clusters is too large. This is usually because the feature vector is not discriminative enough for this type of data, which usually means that more stringent parameters are needed to define whether the data belongs to the same category. Therefore, the clustering results will be further adjusted. The specific steps are as follows:

[0106] Select a representative word and make it the cluster center. The cluster center is determined by the frequency of the business word and the position of the word in the cluster. The determination rule is:

[0107] sigmod(freq)×dist;

[0108] Where freq is the frequency of the business word, and dist is the distance from the business word to the geometric cluster center;

[0109] A threshold t is preset. For clusters that are too large, each data point is iterated to calculate the distance d between the data and the cluster center. The distance d is compared with the threshold t, and data is removed. t is an empirical value, which is set to 0.96 in this implementation.

[0110] If the distance d>threshold t, the data is removed from the cluster.

[0111] Furthermore, in the step of performing secondary clustering:

[0112] After all data are clustered, data selection is performed on clustered data, unclustered data, and data with less than 4 clusters, and secondary clustering is performed after data segmentation;

[0113] For clustered data, select the three data closest to the cluster center;

[0114] Among them, for the data that are not clustered and the number of clusters is less than 4, all such data are selected.

[0115] In this embodiment, considering that after the data is split, related data may be divided into different data sets, therefore, after all data are clustered, for clustered data, three data closest to the cluster center are selected; for data that is not clustered or has less than 4 clusters, all such data are selected. Secondary clustering is performed in the form of steps S300 and S400.

[0116] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0117] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for enterprise label clustering combined with semantic vector fine-tuning, characterized in that: The steps include: Obtain enterprise business scope data, pre-process the data, and calculate the general semantic feature vector; Set up fine-tuning optimization objectives to fine-tune the feature vector and constrain the fine-tuning process; Split the data; The clustering results of kmeans are used to estimate the initial value of DBSCAN, and clustering is performed to obtain the clustering results.

2. The enterprise tag clustering method combined with semantic vector fine-tuning according to claim 1, characterized in that: In the steps of obtaining the enterprise business scope data, preprocessing the data, and calculating the general semantic feature vector: Obtain the enterprise business scope data, perform word segmentation on the enterprise business scope data, and filter the data; Count the word frequency information of tags; Calculate the universal semantic feature vector of each business word; Unify the format of common semantic feature vectors.

3. The enterprise tag clustering method combined with semantic vector fine-tuning according to claim 1, characterized in that: In the step of setting up the fine-tuning optimization objective to fine-tune the feature vector and constraining the fine-tuning process: Set up fine-tuning optimization objectives. The formula for fine-tuning optimization objectives is: Where D is the positive sample space, and D' is the negative sample space; For the positive sample space, W is the current business word, and c is the words around the current business word; For the negative sample space, W is the current business word, and c is a business word with a different meaning from W.

4. The enterprise tag clustering method combined with semantic vector fine-tuning as claimed in claim 3, characterized in that: In the step of setting up the fine-tuning optimization objective to fine-tune the feature vector and constraining the fine-tuning process: The fine-tuning is constrained, and the constraint formula is: Where w, w' are the original vector and the adjusted vector respectively; λ is a hyperparameter.

5. The enterprise tag clustering method combined with semantic vector fine-tuning according to claim 4, characterized in that: After setting up the fine-tuning optimization objective to fine-tune the feature vector and constraining the fine-tuning process: The optimization goal is: L1+L2.

6. The enterprise tag clustering method combined with semantic vector fine-tuning according to claim 1, characterized in that: In the step of estimating the initial value of DBSCAN using the clustering result of kmeans and performing clustering to obtain the clustering result: Roughly estimate the initial value of kmeans based on the data scale and perform clustering; After K-means clustering is completed, the maximum distance from each point in each cluster to the cluster center is counted and arranged in order from low to high, the inflection point information is obtained, the cluster before the inflection point is queried, and data calculation is performed on the queried cluster; the data calculation includes the average number of clusters, the maximum radius of the cluster, the minimum radius of the cluster, the maximum distance of each point in the cluster, and the minimum distance of each point in the cluster; The average number of clusters is used as the MinPts parameter of DBSCAN, and the maximum radius and minimum radius of the cluster are used as the parameter search interval of DBSCAN. Within this interval, with a step size of 0.01, the clustering result with the largest number of clusters is found.

7. The enterprise tag clustering method combined with semantic vector fine-tuning according to claim 6, characterized in that: After the step of finding the clustering result with the largest number of clusters: Adjust the clustering results: select a representative word and make it the cluster center; The cluster center is determined by the frequency of the business word and the position of the word in the cluster. The determination rule is: sigmod(freq)×dist; Where freq is the frequency of the business word, and dist is the distance from the business word to the geometric cluster center; Preset the threshold t, iterate each data point, calculate the distance d between the data and the cluster center, compare the distance d and the threshold t, and remove the data; If the distance d>threshold t, the data is removed from the cluster.

8. The enterprise tag clustering method combined with semantic vector fine-tuning according to claim 7, characterized in that: After using the kmeans clustering results to estimate the initial value of DBSCAN and perform clustering to obtain the clustering results: Perform secondary clustering.

9. The enterprise tag clustering method combined with semantic vector fine-tuning according to claim 7, characterized in that: In the step of secondary clustering: After all data are clustered, data selection is performed on clustered data, non-clustered data, and data with less than 4 clusters, and secondary clustering is performed after data segmentation.

10. The enterprise tag clustering method combined with semantic vector fine-tuning according to claim 9, characterized in that: After all data are clustered, data selection is performed for clustered data, unclustered data, and data with less than 4 clusters. In the step of performing secondary clustering after data segmentation: For clustered data, select the three data closest to the cluster center; For data that is not clustered and has a cluster number less than 4, select all such data.

Images