Data label marking method, device and processor
By performing secondary labeling on data clustering and proportional analysis, the problem of low data label accuracy in the prior art is solved, and the prediction accuracy of unknown data is improved.
Patent Information
- Application Number
- CN202210434729.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-24
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-04-24
AI Technical Summary
In the prior art, the data marked once has low accuracy, resulting in low prediction accuracy for unknown data.
By obtaining the first type of data and the second type of data for preliminary labeling, the data is clustered using a predetermined clustering algorithm, the proportion of data types in the data cluster is determined, and the secondary labeling is performed according to the proportion, and the label value is updated, so as to Improve the accuracy of label data.
Through the secondary labeling process, the accuracy of data labels is improved, thereby improving the prediction accuracy of unknown data.
Smart Images

Figure CN114861780B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular, to a data tagging method, device, computer-readable storage medium, and processor. Background Art
[0002] When clustering data, the data must be marked to obtain labeled data. The labeled data carries the label information. The labeled data with existing label information is trained through a machine learning algorithm to predict unknown data. However, in the process of labeling the data, since the data is labeled once, the labeled data obtained after one labeling is not accurate, and the accuracy of predicting unknown data is low. Summary of the invention
[0003] The main purpose of the present application is to provide a data labeling method, device, computer-readable storage medium and processor to solve the problem of low accuracy of data after one labeling in the prior art.
[0004] According to one aspect of an embodiment of the present invention, a method for marking data tags is provided, comprising: acquiring first type data and second type data, and marking the first type data to obtain a first tag value, and marking the second type data to obtain a second tag value; clustering the first type data and the second type data according to a predetermined clustering algorithm to obtain multiple data clusters, each of the data clusters including at least one of the first type data and at least one of the second type data; determining the proportion of the first type data in each of the data clusters; and based on the proportion, re-marking the first type data and the second type data, so that at least part of the first tag value in each of the data clusters is updated to the second tag value, and at least part of the second tag value is updated to the first tag value.
[0005] Optionally, based on the proportion, the first type of data and the second type of data are secondary marked so that at least part of the first label values in each data cluster are updated to the second label value, and at least part of the second label values are updated to the first label value, including: when the proportion is greater than a proportion threshold, the label value of the second type of data in the current data cluster is updated to the first label value; when the proportion is less than or equal to the proportion threshold, the label value of the first type of data in the current data cluster is updated to the second label value.
[0006] Optionally, when the proportion is greater than a proportion threshold, the label value of the second type of data in the current data cluster is updated to the first label value, including: determining the current data cluster whose proportion is greater than the proportion threshold as a target data cluster; and updating the label values of a predetermined number of the second type of data in the target data cluster to the first label value.
[0007] Optionally, when the proportion is less than or equal to the proportion threshold, the label value of the first type of data in the current data cluster is updated to the second label value, including: determining the current data cluster whose proportion is less than or equal to the proportion threshold as a non-target data cluster; and updating the label values of all the first type of data in the non-target data cluster to the second label value.
[0008] Optionally, when the proportion is greater than a proportion threshold, the label value of the second type of data in the current data cluster is updated to the first label value, including: when the proportions of the first type of data in multiple data clusters are greater than the proportion threshold, comparing the sizes of multiple proportions; and updating the label values of the second type of data in the data clusters to the first label value in order from large to small according to the proportions.
[0009] Optionally, the method further includes: using the first type of data, the first label value, the second type of data and the second label value to train a decision tree model to obtain a tree diagram; determining the rules of the decision tree based on the tree diagram; determining the similarity between the rules of the decision tree and the standard rules; and determining the accuracy of the rules of the decision tree for classification prediction of unknown data based on the similarity.
[0010] Optionally, the dendrogram includes multiple nodes, each of which includes relevant information, and the relevant information includes at least one of the following: node name, data length, data width, Gini coefficient, data volume, function value corresponding to the data, and data name. Determining the rules of the decision tree based on the dendrogram includes: obtaining position information of the target node in the dendrogram; determining the rules of the decision tree corresponding to the target node based on the position information and the relevant information of the target node, and the target node is one of the multiple nodes.
[0011] According to another aspect of an embodiment of the present invention, a data label marking device is also provided, including: a first processing unit, used to obtain first type data and second type data, and mark the first type data to obtain a first label value, and mark the second type data to obtain a second label value; a clustering unit, used to cluster the first type data and the second type data according to a predetermined clustering algorithm to obtain multiple data clusters, each of the data clusters including at least one of the first type data and at least one of the second type data; a first determination unit, used to determine the proportion of the first type data in each of the data clusters; a second processing unit, used to re-mark the first type data and the second type data according to the proportion, so that at least part of the first label value in each of the data clusters is updated to the second label value, and at least part of the second label value is updated to the first label value.
[0012] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein the program executes any one of the methods described.
[0013] According to yet another aspect of the embodiments of the present invention, a processor is provided, wherein the processor is used to run a program, wherein any one of the methods is executed when the program is run.
[0014] In an embodiment of the present invention, firstly, the first type of data and the second type of data are obtained, and the first type of data is marked to obtain a first label value, and the second type of data is marked to obtain a second label value, and then the first type of data and the second type of data are clustered according to a predetermined clustering algorithm to obtain multiple data clusters, each of which includes at least one of the first type of data and at least one of the second type of data, and then the proportion of the first type of data in each of the data clusters is determined, and finally, according to the proportion, the first type of data and the second type of data are re-marked, so that at least part of the first label value in each of the data clusters is updated to the second label value, and at least part of the second label value is updated to the first label value. In this scheme, the first type of data and the second type of data are re-marked according to the proportion of the first type of data in each data cluster. After the data is re-marked, the label data obtained is more accurate, thereby improving the accuracy of predicting unknown data. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0016] Figure 1 A schematic diagram of a flow chart of a data tag marking method according to an embodiment of the present application is shown;
[0017] Figure 2 A schematic diagram showing the proportion of the first type of data in each data cluster;
[0018] Figure 3 A schematic diagram showing the number of first type data in each data cluster at the target node;
[0019] Figure 4 A schematic diagram of a dendrogram is shown;
[0020] Figure 5 A schematic structural diagram of a data tag marking device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0024] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element, or there may be intermediate elements. Moreover, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element, or "connected" to the other element through a third element.
[0025] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:
[0026] Clustering: Clustering is a machine learning technique that involves grouping data points. Given a set of data points, clustering can be used to divide each data point into a specific group. In theory, data points in the same group should have similar attributes and / or similar features, while data points in different groups should have different attributes and / or features with lower similarity. Clustering is an unsupervised learning method and a commonly used statistical data analysis technique in many fields. The most common algorithm in clustering algorithms is prototype clustering.
[0027] Prototype clustering: Prototype clustering is also called "prototype-based clustering". The prototype clustering algorithm assumes that the clustering structure can be described by a set of prototypes. The algorithm first initializes the prototype and then iteratively updates and solves the prototype. Different prototype representations and different solving methods will produce different algorithms. The more typical algorithm is KMeans.
[0028] KMeans algorithm: first randomly select K objects as the initial cluster centers, then calculate the distance between each object and each initial cluster center, assign each object to the initial cluster center with the closest distance, the initial cluster center and the assigned objects form a cluster, once all objects are assigned, the cluster center of each cluster will be recalculated according to the existing objects in the cluster, this process will be repeated until the termination condition is met, the termination condition includes at least one of the following: no (or the minimum number) objects are reassigned to different cluster centers, no (or the minimum number) cluster centers change again, the error sum of squares is locally minimum (the error sum of squares refers to the sum of squares of the differences between all samples and the nearest cluster center, Among them, C i represents the i-th cluster, x represents a point in the cluster (an object in the cluster), u i is the center of the i-th cluster);
[0029] MinibatchKMeans algorithm: It is evolved from the KMeans algorithm. It mainly solves the problem of long calculation time of KMeans under large data volume. The MinibatchKMeans algorithm uses a method called Mini Batch (batch processing) to calculate the distance between data points. It extracts a part of samples from different categories to represent their respective types for calculation. It is not necessary to use all data samples, thereby reducing the corresponding running time.
[0030] As mentioned in the background technology, the accuracy of data marked once in the prior art is low. In order to solve the above problem, in a typical embodiment of the present application, a data labeling method, device, computer-readable storage medium and processor are provided.
[0031] According to an embodiment of the present application, a data tagging method is provided.
[0032] Figure 1 FIG. 1 is a flow chart of a data tagging method according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0033] Step S101, obtaining first type data and second type data, and marking the first type data to obtain a first label value, and marking the second type data to obtain a second label value;
[0034] Step S102, clustering the first type of data and the second type of data according to a predetermined clustering algorithm to obtain a plurality of data clusters, each of which includes at least one of the first type of data and at least one of the second type of data;
[0035] Step S103, determining the proportion of the first type of data in each of the data clusters;
[0036] Step S104, re-labeling the first type data and the second type data according to the proportions, so that at least part of the first label values in each of the data clusters are updated to the second label values, and at least part of the second label values are updated to the first label values.
[0037] In the above method, firstly, the first type of data and the second type of data are obtained, and the first type of data is marked to obtain the first label value, and the second type of data is marked to obtain the second label value, and then the first type of data and the second type of data are clustered according to a predetermined clustering algorithm to obtain multiple data clusters, each of which includes at least one of the first type of data and at least one of the second type of data, and then the proportion of the first type of data in each of the data clusters is determined, and finally, according to the proportion, the first type of data and the second type of data are re-marked, so that at least part of the first label value in each of the data clusters is updated to the second label value, and at least part of the second label value is updated to the first label value. In this scheme, the first type of data and the second type of data are re-marked according to the proportion of the first type of data in each data cluster. After the data is re-marked, the label data obtained is more accurate, thereby improving the accuracy of predicting unknown data.
[0038] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0039] Specifically, the first type of data and the second type of data can be marked for the first time according to pre-established rules. The first label value can be set to 1 and the second label value can be set to 0. Of course, this is not limited to the above situation. Technical personnel in this field can also select appropriate first label value and second label value according to actual conditions.
[0040] In one embodiment, the proportion of the first type of data in each of the data clusters is determined, for example Figure 2 As shown, there are 4 data clusters in total. The data volume of the first data cluster is 600, and the proportion of the first type of data in the first data cluster is 20%; the data volume of the second data cluster is 300, and the proportion of the first type of data in the second data cluster is 70%; the data volume of the third data cluster is 400, and the proportion of the first type of data in the third data cluster is 90%; the data volume of the fourth data cluster is 700, and the proportion of the first type of data in the fourth data cluster is 10%.
[0041] It should be noted that the predetermined clustering algorithm may be the MinibatchKMeans algorithm, but is certainly not limited to the MinibatchKMeans algorithm, and those skilled in the art may also select any other feasible clustering algorithm.
[0042] In one embodiment of the present application, the first type of data and the second type of data are secondary labeled according to the above-mentioned proportion, so that at least part of the first label value in each of the above-mentioned data clusters is updated to the second label value, and at least part of the second label value is updated to the first label value, including: when the above-mentioned proportion is greater than the proportion threshold, the label value of the second type of data in the current data cluster is updated to the first label value; when the above-mentioned proportion is less than or equal to the above-mentioned proportion threshold, the label value of the first type of data in the current data cluster is updated to the second label value. In this embodiment, the first type of data and the second type of data can be secondary labeled more accurately according to the size relationship between the proportion and the proportion threshold, thereby further ensuring that the obtained label data is more accurate.
[0043] In a specific embodiment, the ratio threshold is 80%. Figure 2 Among the four data clusters, only the proportion of the first type of data in the third data cluster is greater than 80%, and the label value of the second type of data in the third data cluster is updated to the first label value.
[0044] In another embodiment of the present application, when the above-mentioned proportion is greater than the proportion threshold, the label value of the above-mentioned second type of data in the current data cluster is updated to the above-mentioned first label value, including: determining the current data cluster whose proportion is greater than the above-mentioned proportion threshold as the target data cluster; updating the label value of the above-mentioned second type of data of a predetermined number in the above-mentioned target data cluster to the above-mentioned first label value. In this embodiment, the label value of the second type of data of a predetermined number can be updated according to business needs, so that the obtained data cluster can be guaranteed to better meet business needs.
[0045] In another specific embodiment, part of the first type of data and part of the second type of data in the data cluster may be extracted. For example, 2000 data are extracted, and the proportion of the first type of data is 1 / 50. If the amount of the first type of data needs to be updated to twice the original amount, then the label values of 2000×(1 / 50)=40 second type of data need to be updated to the first label value, such as Figure 2 The third data cluster in can meet the update requirement of the predetermined number of 400×(1-0.9)=40.
[0046] In another embodiment of the present application, when the proportion is less than or equal to the proportion threshold, the label value of the first type of data in the current data cluster is updated to the second label value, including: determining the current data cluster whose proportion is less than or equal to the proportion threshold as a non-target data cluster; updating the label value of all the first type of data in the non-target data cluster to the second label value. In this embodiment, when the proportion is less than or equal to the proportion threshold, the label value of all the first type of data in the data cluster can be updated to the second label value, which further ensures that the label value of all the data in the data cluster is the second label value, thereby further ensuring that the obtained label data is more accurate.
[0047] In another embodiment of the present application, when the above-mentioned proportion is greater than the proportion threshold, the label value of the above-mentioned second type of data in the current data cluster is updated to the above-mentioned first label value, including: when the above-mentioned proportion of the above-mentioned first type of data in multiple data clusters is greater than the above-mentioned proportion threshold, compare the sizes of multiple above-mentioned proportions; and update the label values of the above-mentioned second type of data in the above-mentioned data cluster to the above-mentioned first label value in order from large to small according to the above-mentioned proportions. In this embodiment, the label value is updated in a predetermined order. The larger the proportion of the first type of data in the data cluster, the better the controllability of the data cluster, and the more accurate the number of the first type of data in the data cluster. Therefore, the data cluster with a large proportion of the first type of data is updated first, so that the data can be more accurately re-labeled.
[0048] In a specific embodiment of the present application, the method further includes: using the first type of data, the first label value, the second type of data and the second label value to train the decision tree model to obtain a tree diagram; determining the rules of the decision tree according to the tree diagram; determining the similarity between the rules of the decision tree and the standard rules; and determining the accuracy of the rules of the decision tree for classifying and predicting unknown data according to the similarity. In this embodiment, it is possible to further verify whether the rules of the decision tree that have been labeled twice are reasonable. The higher the similarity, the higher the accuracy of the rules of the decision tree for classifying and predicting unknown data.
[0049] In another specific embodiment of the present application, the above-mentioned tree diagram includes multiple nodes, each of the above-mentioned nodes includes relevant information, and the above-mentioned relevant information includes at least one of the following: node name, data length, data width, Gini coefficient, data volume, function value corresponding to the data, data name, and determining the rules of the decision tree according to the above-mentioned tree diagram includes: obtaining the position information of the target node in the above-mentioned tree diagram; determining the rule of the above-mentioned decision tree corresponding to the above-mentioned target node according to the above-mentioned position information and the above-mentioned relevant information of the above-mentioned target node, and the above-mentioned target node is one of the multiple above-mentioned nodes. In this embodiment, the rules of the decision tree can be determined more accurately according to the relevant information, thereby further improving the accuracy of the decision tree rules in classifying and predicting unknown data.
[0050] In another specific embodiment, Figure 2 The label value of the first type of data is 1, the label value of the second type of data is 0, the ratio threshold is 80%, and only the proportion of the first type of data in the third data cluster is greater than 80%. The label value of the second type of data in the third data cluster is updated to 1, and the label values of the first type of data in the first data cluster, the second data cluster, and the fourth data cluster are updated to 0. The decision tree model is trained, and the resulting tree diagram is Figure 3 The left picture in the middle, Figure 3 The middle figure is a fan diagram of the proportion of the first label value of the first type of data and the second label value of the second type of data. The decision tree rules are obtained according to the tree diagram. The data that can be secondary labeled in the third data cluster fall on different target nodes, for example, they fall on the target nodes node#2, node#5, node#7 and node#10 respectively. There are 34 first-type data in node#2, 3 first-type data in node#5, 2 first-type data in node#7, and 1 first-type data in node#10. The decision tree model is trained with the iris data set (a data set consisting of 50 groups of data from three types of irises). The decision tree rules are as follows: Figure 4 As shown, the graph includes 17 target nodes, where the node name is node, the data length is petal length (cm), the data width is petal width (cm), the Gini coefficient is gini, the data volume is samples, the function value corresponding to the data is value, the data name is class, and the relevant information of each node is as follows:
[0051] node#0: petal length(cm)≤2.45, gini=0.667, samples=150, value=[50,50,50], class=setosa;
[0052] node#1:gini=0.0,samples=50,value=[50,0,0],class=setosa;
[0053] node#2:petal width(cm)≤1.75,gini=0.5,samples=100,value=[0,50,50],class=versicolor;
[0054] node#3:petal length(cm)≤4.95,gini=0.168,samples=54,value=[0,49,50],class=versicolor;
[0055] node#4:petal width(cm)≤1.65,gini=0.041,samples=48,value=[0,47,1],class=versicolor;
[0056] node#5:gini=0.0,samples=47,value=[0,47,0],class=versicolor;
[0057] node#6:gini=0.0,samples=1,value=[0,0,1],class=virginica;
[0058] node#7:petal width(cm)≤1.55,gini=0.444,samples=6,value=[0,2,4],class=virginica;
[0059] node#8:gini=0.0,samples=3,value=[0,0,3],class=virginica;
[0060] node#9:petal length(cm)≤6.95,gini=0.444,samples=3,value=[0,2,1],class=versicolor;
[0061] node#10:gini=0.0,samples=2,value=[0,2,0],class=versicolor;
[0062] node#11: gini=0.0, samples=1, value=[0,0,1], class=virginica;
[0063] node#12: petal length(cm)≤4.85, gini=0.043, samples=46, value=[0,1,45], class=virginica;
[0064] node#13: petal length(cm)≤5.95, gini=0.444, samples=3, value=[0,1,2], class=virginica;
[0065] node#14: gini=0.0, samples=43, value=[0,1,0], class=versicolor;
[0066] node#15: gini=0.0, samples=2, value=[0,0,2], class=virginica;
[0067] node#16: gini=0.0, samples=43, value=[0,0,43], class=virginica.
[0068] The present application also provides a data tag marking device. It should be noted that the data tag marking device of the present application can be used to execute the data tag marking method provided in the present application. The data tag marking device provided in the present application is introduced below.
[0069] Figure 5 Schematic diagram of a data tag marking device according to an embodiment of the present application. Figure 5 As shown, the device comprises:
[0070] The first processing unit 10 is used to obtain the first type of data and the second type of data, and mark the first type of data to obtain a first label value, and mark the second type of data to obtain a second label value;
[0071] A clustering unit 20, configured to cluster the first type of data and the second type of data according to a predetermined clustering algorithm to obtain a plurality of data clusters, each of which includes at least one of the first type of data and at least one of the second type of data;
[0072] A first determining unit 30, used to determine the proportion of the first type of data in each of the data clusters;
[0073] The second processing unit 40 is used to perform secondary marking on the above-mentioned first type data and the above-mentioned second type data according to the above-mentioned proportion, so that at least part of the above-mentioned first label values in each of the above-mentioned data clusters are updated to the above-mentioned second label values, and at least part of the above-mentioned second label values are updated to the above-mentioned first label values.
[0074] In the above-mentioned device, the first processing unit obtains the first type of data and the second type of data, and marks the first type of data to obtain a first label value, marks the second type of data to obtain a second label value, the clustering unit clusters the first type of data and the second type of data according to a predetermined clustering algorithm to obtain a plurality of data clusters, each of which includes at least one of the first type of data and at least one of the second type of data, the first determination unit determines the proportion of the first type of data in each of the data clusters, and the second processing unit performs secondary labeling on the first type of data and the second type of data according to the proportion, so that at least part of the first label value in each of the data clusters is updated to the second label value, and at least part of the second label value is updated to the first label value. In this scheme, the first type of data and the second type of data are secondary labeled according to the proportion of the first type of data in each data cluster. After the data is secondary labeled, the label data obtained is more accurate, thereby improving the accuracy of predicting unknown data.
[0075] Specifically, the first type of data and the second type of data can be marked for the first time according to pre-established rules. The first label value can be set to 1 and the second label value can be set to 0. Of course, this is not limited to the above situation. Technical personnel in this field can also select appropriate first label value and second label value according to actual conditions.
[0076] It should be noted that the predetermined clustering algorithm may be the MinibatchKMeans algorithm, but is certainly not limited to the MinibatchKMeans algorithm, and those skilled in the art may also select any other feasible clustering algorithm.
[0077] In one embodiment of the present application, the second processing unit includes a first processing module and a second processing module. The first processing module is used to update the label value of the second type of data in the current data cluster to the first label value when the proportion is greater than the proportion threshold; the second processing module is used to update the label value of the first type of data in the current data cluster to the second label value when the proportion is less than or equal to the proportion threshold. In this embodiment, the first type of data and the second type of data can be more accurately marked according to the size relationship between the proportion and the proportion threshold, thereby further ensuring that the obtained label data is more accurate.
[0078] In another embodiment of the present application, the first processing module includes a first processing submodule and a second processing submodule, the first processing submodule is used to determine the current data cluster whose proportion is greater than the proportion threshold as the target data cluster; the second processing submodule is used to update the label value of the predetermined number of the second type of data in the target data cluster to the first label value. In this embodiment, the label value of the predetermined number of the second type of data can be updated according to business needs, so that the obtained data cluster can be guaranteed to better meet business needs.
[0079] In another embodiment of the present application, the second processing module includes a third processing submodule and a fourth processing submodule, the third processing submodule is used to determine the current data cluster whose proportion is less than or equal to the proportion threshold as a non-target data cluster; the fourth processing submodule is used to update the label value of all the first type data in the non-target data cluster to the second label value. In this embodiment, when the proportion is less than or equal to the proportion threshold, the label value of all the first type data in the data cluster can be updated to the second label value, which further ensures that the label value of all the data in the data cluster is the second label value, thereby further ensuring that the obtained label data is more accurate.
[0080] In another embodiment of the present application, the first processing module includes a comparison submodule and a fifth processing submodule, the comparison submodule is used to compare the sizes of the above-mentioned proportions when the above-mentioned proportions of the above-mentioned first type of data in the above-mentioned multiple data clusters are all greater than the above-mentioned proportion threshold; the fifth processing submodule is used to update the label values of the above-mentioned second type of data in the above-mentioned data cluster to the above-mentioned first label value in order from large to small according to the above-mentioned proportions. In this embodiment, the label value is updated in a predetermined order, the larger the proportion of the first type of data in the data cluster, the better the controllability of the data cluster, and the more accurate the number of the first type of data in the data cluster, therefore, the data cluster with a large proportion of the first type of data is updated first, so that the data can be more accurately re-labeled.
[0081] In a specific embodiment of the present application, the above-mentioned device also includes a training unit, a second determination unit, a third determination unit and a fourth determination unit. The training unit is used to train the decision tree model using the above-mentioned first type of data, the above-mentioned first label value, the above-mentioned second type of data and the above-mentioned second label value to obtain a tree diagram; the second determination unit is used to determine the rules of the decision tree according to the above-mentioned tree diagram; the third determination unit is used to determine the similarity between the rules of the above-mentioned decision tree and the standard rules; the fourth determination unit is used to determine the accuracy of the rules of the above-mentioned decision tree for classification prediction of unknown data according to the above-mentioned similarity. In this embodiment, it can be further verified whether the rules of the decision tree that have been marked twice are reasonable. The higher the similarity, the higher the accuracy of the rules of the decision tree for classification prediction of unknown data.
[0082] In another specific embodiment of the present application, the above-mentioned tree diagram includes multiple nodes, each of the above-mentioned nodes includes relevant information, and the above-mentioned relevant information includes at least one of the following: node name, data length, data width, Gini coefficient, data volume, function value corresponding to the data, data name, and the second determination unit includes an acquisition module and a determination module, and the acquisition module is used to obtain the position information of the target node in the above-mentioned tree diagram; the determination module is used to determine the rule of the above-mentioned decision tree corresponding to the above-mentioned target node according to the above-mentioned position information and the above-mentioned relevant information of the above-mentioned target node, and the above-mentioned target node is one of the multiple above-mentioned nodes. In this embodiment, the rules of the decision tree can be determined more accurately according to the relevant information, thereby further improving the accuracy of the classification prediction of unknown data by the rules of the decision tree.
[0083] The marking device for the above-mentioned data label includes a processor and a memory. The above-mentioned first processing unit, clustering unit, first determination unit and second processing unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.
[0084] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the accuracy of predicting unknown data can be improved by adjusting kernel parameters.
[0085] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0086] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned data tagging method is implemented.
[0087] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data tag marking method is executed when the program is running.
[0088] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:
[0089] Step S101, obtaining first type data and second type data, and marking the first type data to obtain a first label value, and marking the second type data to obtain a second label value;
[0090] Step S102, clustering the first type of data and the second type of data according to a predetermined clustering algorithm to obtain a plurality of data clusters, each of which includes at least one of the first type of data and at least one of the second type of data;
[0091] Step S103, determining the proportion of the first type of data in each of the data clusters;
[0092] Step S104, re-labeling the first type data and the second type data according to the proportions, so that at least part of the first label values in each of the data clusters are updated to the second label values, and at least part of the second label values are updated to the first label values.
[0093] The devices in this article can be servers, PCs, PADs, mobile phones, etc.
[0094] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:
[0095] Step S101, obtaining first type data and second type data, and marking the first type data to obtain a first label value, and marking the second type data to obtain a second label value;
[0096] Step S102, clustering the first type of data and the second type of data according to a predetermined clustering algorithm to obtain a plurality of data clusters, each of which includes at least one of the first type of data and at least one of the second type of data;
[0097] Step S103, determining the proportion of the first type of data in each of the data clusters;
[0098] Step S104, re-labeling the first type data and the second type data according to the proportions, so that at least part of the first label values in each of the data clusters are updated to the second label values, and at least part of the second label values are updated to the first label values.
[0099] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0100] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above-mentioned units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0101] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0102] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0103] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0104] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:
[0105] 1) The data labeling method of the present application first obtains the first type of data and the second type of data, and labels the first type of data to obtain the first label value, labels the second type of data to obtain the second label value, and then clusters the first type of data and the second type of data according to a predetermined clustering algorithm to obtain multiple data clusters, each of which includes at least one of the first type of data and at least one of the second type of data, and then determines the proportion of the first type of data in each of the data clusters. Finally, according to the proportion, the first type of data and the second type of data are relabeled, so that at least part of the first label value in each of the data clusters is updated to the second label value, and at least part of the second label value is updated to the first label value. In this scheme, the first type of data and the second type of data are relabeled according to the proportion of the first type of data in each data cluster. After the data is relabeled, the label data obtained is more accurate, thereby improving the accuracy of predicting unknown data.
[0106] 2) The data labeling device of the present application, the first processing unit obtains the first type of data and the second type of data, and labels the first type of data to obtain the first label value, labels the second type of data to obtain the second label value, the clustering unit clusters the first type of data and the second type of data according to a predetermined clustering algorithm to obtain multiple data clusters, each of which includes at least one of the first type of data and at least one of the second type of data, the first determination unit determines the proportion of the first type of data in each of the data clusters, and the second processing unit performs secondary labeling on the first type of data and the second type of data according to the proportion, so that at least part of the first label value in each of the data clusters is updated to the second label value, and at least part of the second label value is updated to the first label value. In this scheme, the first type of data and the second type of data are secondary labeled according to the proportion of the first type of data in each data cluster. After the data is secondary labeled, the obtained label data is more accurate, thereby improving the accuracy of predicting unknown data.
[0107] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A data labeling method, It is characterized in that include: Acquire first type data and second type data, and mark the first type data to obtain a first label value, and mark the second type data to obtain a second label value; Clustering the first type of data and the second type of data according to a predetermined clustering algorithm to obtain a plurality of data clusters, each of the data clusters including at least one of the first type of data and at least one of the second type of data; Determining the proportion of the first type of data in each of the data clusters; According to the proportion, the first type of data and the second type of data are re-labeled, so that at least part of the first label value in each of the data clusters is updated to the second label value, and at least part of the second label value is updated to the first label value; According to the proportion, the first type of data and the second type of data are secondary marked so that at least part of the first label value in each of the data clusters is updated to the second label value, and at least part of the second label value is updated to the first label value, including: When the proportion is greater than a proportion threshold, updating the label value of the second type of data in the current data cluster to the first label value; When the proportion is less than or equal to the proportion threshold, the label value of the first type of data in the current data cluster is updated to the second label value.
2. The method according to claim 1, It is characterized in that When the proportion is greater than a proportion threshold, updating the label value of the second type of data in the current data cluster to the first label value includes: Determine the current data cluster whose proportion is greater than the proportion threshold as a target data cluster; The label values of a predetermined number of the second type data in the target data cluster are updated to the first label values.
3. The method according to claim 1, It is characterized in that When the proportion is less than or equal to the proportion threshold, updating the label value of the first type of data in the current data cluster to the second label value includes: Determine the current data cluster whose proportion is less than or equal to the proportion threshold as a non-target data cluster; Update the label values of all the first type data in the non-target data cluster to the second label value.
4. The method according to claim 1, It is characterized in that When the proportion is greater than a proportion threshold, updating the label value of the second type of data in the current data cluster to the first label value includes: When the proportion of the first type of data in the plurality of data clusters is greater than the proportion threshold, comparing the sizes of the plurality of proportions; The label values of the second type of data in the data cluster are updated to the first label values in order from large to small according to the proportions.
5. The method according to any one of claims 1 to 4, It is characterized in that The method further comprises: Using the first type of data, the first label value, the second type of data, and the second label value, training a decision tree model to obtain a tree diagram; Determine a decision tree rule according to the tree diagram; Determining the similarity between the rules of the decision tree and the standard rules; The accuracy of the classification prediction of unknown data by the decision tree rule is determined according to the similarity.
6. The method according to claim 5, It is characterized in that The dendrogram includes a plurality of nodes, each of which includes relevant information, and the relevant information includes at least one of the following: node name, data length, data width, Gini coefficient, data volume, function value corresponding to the data, and data name. The decision tree rule is determined according to the dendrogram, including: Obtaining the position information of the target node in the tree diagram; A rule of the decision tree corresponding to the target node is determined according to the location information and the relevant information of the target node, wherein the target node is one of the multiple nodes.
7. A data tag marking device, It is characterized in that include: A first processing unit is used to obtain first type data and second type data, and mark the first type data to obtain a first label value, and mark the second type data to obtain a second label value; a clustering unit, configured to cluster the first type of data and the second type of data according to a predetermined clustering algorithm to obtain a plurality of data clusters, each of the data clusters including at least one data of the first type and at least one data of the second type; A first determining unit, used to determine the proportion of the first type of data in each of the data clusters; A second processing unit is used to perform secondary labeling on the first type of data and the second type of data according to the proportion, so that at least part of the first label values in each of the data clusters are updated to the second label values, and at least part of the second label values are updated to the first label values; Wherein, the second processing unit includes: A first processing module, configured to update the label value of the second type of data in the current data cluster to the first label value when the proportion is greater than a proportion threshold; The second processing module is used to update the label value of the first type of data in the current data cluster to the second label value when the proportion is less than or equal to the proportion threshold.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein the program executes the method of any one of claims 1 to 6.
