Incremental clustering method, device, equipment and storage medium based on group attributes

Abstract

The present invention belongs to the technical field of big data processing, and specifically provides an incremental clustering method, device, equipment and storage medium based on group attributes, aiming to solve the problem of how to deal with suspected merged result pairs in incremental clustering results. To this end, the method of the present invention includes: acquiring newly added data and historical data, the data sample contains group attributes, and the group attributes include group name and team name; performing incremental clustering in accordance with group attribute constraints to obtain second clustering data and / or suspected merged data; following the group attribute constraints, manually intervening to process the suspected merged data pairs arranged by priority to obtain third clustering data; updating the first clustering data according to the second clustering data and the third clustering data. Applying the method of the present invention, the merging of cluster errors is suppressed, the correct merging is maintained, and the clustering accuracy is improved; at the same time, the timeliness and influence of manual confirmation information are enhanced, so that it can be persistent, and the purpose of reinforcement learning is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and specifically provides an incremental clustering method, device, equipment and storage medium based on group attributes. Background Art

[0002] Clustering is the process of classifying data into different classes or clusters. Cluster analysis is an exploratory analysis. In the classification process, people do not need to give a classification standard in advance. Cluster analysis can automatically classify based on sample data. Cluster analysis has been widely used in commercial data analysis, pattern recognition, image processing and other fields.

[0003] How to efficiently obtain information from massive data has become the focus of clustering algorithm research. In the current clustering algorithm, with the increase of data scale and the increase of data distribution difficulty, the probability of high-scoring negative examples will also increase, which seriously affects the accuracy of clustering and leads to merging errors in clustering results. When confirming the suspected merged results in the clustering results of incremental clustering, if the merge or non-merge processing is simply performed based on the confirmation results, although the clustering accuracy can be increased, the non-merged information is simply discarded and its information is not inherited and continued, and it cannot generate more use value, which has a great impact on the accuracy and efficiency of image classification and recognition in practical applications such as image recognition and pattern recognition. Therefore, how to effectively solve the problems of short timeliness, weak impact and inability to persist when confirming the suspected merged results generated in the incremental clustering process, and improve the accuracy and efficiency of image classification and recognition, has become an urgent problem to be solved in this field.

[0004] Accordingly, a new solution is needed in the art to solve the above problems. Summary of the invention

[0005] The present invention aims to solve the above technical problems, namely, to solve the problems of short timeliness, weak impact, and inability to persist when confirming suspected merging results generated in the incremental clustering process, resulting in low accuracy and efficiency in image classification and recognition.

[0006] In a first aspect, the present invention provides an incremental clustering method based on group attributes, the method comprising:

[0007] S1, obtaining data to be processed, wherein the data to be processed includes newly added data and first clustering data, wherein the first clustering data is historical data for which clustering has been completed, and the data type of the data to be processed includes image data or text data, and each sample in the data to be processed includes group attributes, and the group attributes include a group number and a team number;

[0008] S2, performing incremental clustering on the data to be processed in accordance with the group attribute constraint to obtain a clustering result, wherein the clustering result includes the second clustering data and / or the suspected merged data that have completed the incremental clustering, wherein the suspected merged data includes one or more suspected merged data pairs, wherein the suspected merged data pairs include two merged objects and a similarity score between the two merged objects, and each merged object includes a label and at least one sample;

[0009] S3, updating the first clustering data according to the second clustering data;

[0010] S4, performing candidate merging queue management on the suspected merged data to obtain a processing priority queue of the suspected merged data pair;

[0011] S5, in response to the preset execution condition, continue to execute step S1 or enter the suspected merged data processing flow.

[0012] In one implementation of the above-mentioned incremental clustering method based on group attributes, the steps of the "suspected merged data processing flow" specifically include:

[0013] Pause incremental clustering operations;

[0014] Obtaining the suspected merged data pair with the highest priority in the processing priority queue, and discriminating the suspected merged data pair with the highest priority to obtain a category discrimination result, wherein the category discrimination result includes one of an impure category, an unmerged category, or a merged category;

[0015] According to the category discrimination result and in compliance with the group attribute constraint, a third category of clustering data is obtained;

[0016] Updating the first clustering data according to the third clustering data;

[0017] Resume the incremental clustering operation and execute step S1 to start new incremental clustering.

[0018] In one embodiment of the above-mentioned incremental clustering method based on group attributes, the step of "obtaining the third type of clustering data according to the category discrimination result and following the group attribute constraint" specifically includes:

[0019] When the category discrimination result is an impure category, performing group attribute splitting on the samples of the suspected merged data pair;

[0020] When the category discrimination result is a non-merging category, assigning a first group of attributes to the group attributes of the sample of the suspected merged data pair;

[0021] When the discrimination result is a merge category, a group attribute constraint check is performed on the samples of the suspected merged data pair, and it is determined whether to merge the suspected merged data pair according to the result of the group attribute constraint check.

[0022] In one embodiment of the above-mentioned incremental clustering method based on group attributes, the step of "when the category discrimination result is an impure category, performing group attribute splitting on the samples of the suspected merged data pair" specifically includes:

[0023] Classify the samples in the suspected merged data pairs with the same group number and team number into a first split class;

[0024] Classify the remaining samples in the suspected merged data pairs into a second split class;

[0025] New labels are set for the first split class and the second split class respectively.

[0026] In one embodiment of the above-mentioned incremental clustering method based on group attributes, the step of "when the category discrimination result is not to merge the categories, assigning a first group of attributes to the group attributes of the samples of the suspected merged data pair" specifically includes:

[0027] Assigning a new group number with the same value to the samples in the suspected merged data pair;

[0028] Team numbers with different values ​​are respectively assigned to samples corresponding to the two types of labels in the suspected merged data pair.

[0029] In one embodiment of the above-mentioned incremental clustering method based on group attributes, the step of "when the discrimination result is a merge category, performing a group attribute constraint check on the samples of the suspected merged data pair, and determining whether to merge the suspected merged data pair according to the result of the group attribute constraint check" specifically includes:

[0030] Checking whether the group attributes of all samples in the suspected merged data pair conflict;

[0031] If there is a conflict, it will not be merged;

[0032] If there is no conflict, assigning a second group of attributes to the group attributes of the samples in the suspected merged data pair;

[0033] According to the second set of attribute values, merging the suspected merged data pairs;

[0034] The step of “assigning a second group of attributes to the group attributes of the samples in the suspected merged data pair” specifically includes:

[0035] Allocating a new group number with the same value to the samples of the suspected merged data pair;

[0036] The team numbers with the same values ​​are assigned to the samples of the suspected merged data pairs.

[0037] In one embodiment of the above-mentioned incremental clustering method based on group attributes, the step of "checking whether the group attributes of all samples in the suspected merged data pair conflict" specifically includes:

[0038] Select all samples whose group number is not 0;

[0039] When the group numbers are the same and the team numbers are the same, the group attributes do not conflict;

[0040] When the group numbers are the same but the team numbers are different, the group attributes conflict.

[0041] In one embodiment of the above-mentioned incremental clustering method based on group attributes, the group attribute constraint method includes:

[0042] The sample with group number 0 is an invalid group attribute and has no constraint relationship with any other clustering data;

[0043] The samples whose group number is not 0 have a valid group attribute;

[0044] The samples with the same group number and team number have the same label;

[0045] The samples with the same group number but different team numbers have different labels;

[0046] There is no constraint relationship between the samples with different group numbers.

[0047] In one embodiment of the above-mentioned incremental clustering method based on group attributes, the step of "responding to a preset execution condition, continuing to execute step S1 or entering the suspected merge data processing flow" specifically includes:

[0048] When the number of suspected merged data pairs obtained in step S2 reaches a preset number threshold, or the cumulative statistical incremental clustering completion time in step S2 reaches a preset time threshold, the suspected merged data processing flow is entered.

[0049] In a second aspect, the present invention provides an incremental clustering device based on group attributes, the device comprising:

[0050] A data loading module, wherein the data acquisition module is configured to acquire data to be processed, wherein the data to be processed includes newly added data and first clustering data, wherein the first clustering data is historical data for which clustering has been completed, and the data type of the data to be processed includes image data or text data, and each sample in the data to be processed includes group attributes, wherein the group attributes include a group number and a team number;

[0051] An incremental clustering module, wherein the incremental clustering module is configured to perform incremental clustering on the data to be processed in accordance with the group attribute constraint to obtain a clustering result, wherein the clustering result includes the second clustering data for which incremental clustering has been completed and the suspected merged data, wherein the suspected merged data includes one or more suspected merged data pairs, wherein the suspected merged data pairs include two merged objects and a similarity score between the two merged objects, and each merged object includes a label and at least one sample;

[0052] a historical data maintenance module, wherein the historical data maintenance module is configured to update the first clustering data according to the second clustering data;

[0053] a suspected merged data management module, wherein the suspected merged data management module is configured to perform candidate merge queue management on the suspected merged data to obtain a processing priority queue of the suspected merged data pairs;

[0054] A human-computer interaction determination module is configured to, in response to a preset execution condition, continue to execute the incremental clustering process or enter the suspected merge data processing flow.

[0055] In one embodiment of the above-mentioned incremental clustering device based on group attributes, the human-computer interaction identification module is further configured to perform the following operations:

[0056] Pause incremental clustering operations;

[0057] Obtaining the suspected merged data pair with the highest priority in the processing priority queue, and discriminating the suspected merged data pair with the highest priority to obtain a category discrimination result, wherein the category discrimination result includes one of an impure category, an unmerged category, or a merged category;

[0058] According to the category discrimination result and in compliance with the group attribute constraint, a third category of clustering data is obtained;

[0059] By means of a historical data maintenance module, the first clustering data is updated according to the third clustering data;

[0060] Resume the incremental clustering process and start a new incremental clustering.

[0061] In one embodiment of the above-mentioned incremental clustering device based on group attributes, the human-computer interaction identification module is further configured to perform the following operations:

[0062] When the category discrimination result is that the category is an impure category, performing group attribute splitting on the samples of the suspected merged data pair;

[0063] When the category discrimination result is a non-merging category, assigning a first group of attributes to the group attributes of the sample of the suspected merged data pair;

[0064] When the discrimination result is a merge category, a group attribute constraint check is performed on the samples of the suspected merged data pair, and it is determined whether to merge the suspected merged data pair according to the result of the group attribute constraint check.

[0065] In one embodiment of the above-mentioned incremental clustering device based on group attributes, the human-computer interaction identification module specifically performs the following operations:

[0066] Classify the samples in the suspected merged data pairs with the same group number and team number into a first split class;

[0067] Classify the remaining samples in the suspected merged data pairs into a second split class;

[0068] New labels are set for the first split class and the second split class respectively.

[0069] In one embodiment of the above-mentioned incremental clustering device based on group attributes, the human-computer interaction identification module specifically performs the following operations:

[0070] Assigning a new group number with the same value to the samples in the suspected merged data pair;

[0071] Team numbers with different values ​​are respectively assigned to samples corresponding to the two types of labels in the suspected merged data pair.

[0072] In one embodiment of the above-mentioned incremental clustering device based on group attributes, the human-computer interaction identification module specifically performs the following operations:

[0073] Checking whether the group attributes of all samples in the suspected merged data pair conflict;

[0074] If there is a conflict, it will not be merged;

[0075] If there is no conflict, assigning a second group of attributes to the group attributes of the samples in the suspected merged data pair;

[0076] According to the second set of attribute values, merging the suspected merged data pairs;

[0077] The step of “assigning a second group of attributes to the group attributes of the samples in the suspected merged data pair” specifically includes:

[0078] Allocating a new group number with the same value to the samples of the suspected merged data pair;

[0079] The team numbers with the same values ​​are assigned to the samples of the suspected merged data pairs.

[0080] In one embodiment of the above-mentioned incremental clustering device based on group attributes, the human-computer interaction identification module specifically performs the following operations:

[0081] Select all samples whose group number is not 0;

[0082] When the group numbers are the same and the team numbers are the same, the group attributes do not conflict;

[0083] When the group numbers are the same but the team numbers are different, the group attributes conflict.

[0084] In one embodiment of the above-mentioned incremental clustering device based on group attributes, the group attribute constraint method includes:

[0085] The sample with group number 0 is an invalid group attribute and has no constraint relationship with any other clustering data;

[0086] The samples whose group number is not 0 have a valid group attribute;

[0087] The samples with the same group number and team number have the same label;

[0088] The samples with the same group number but different team numbers have different labels;

[0089] There is no constraint relationship between the samples with different group numbers.

[0090] In one implementation of the above-mentioned incremental clustering device based on group attributes, the suspected merged data management module specifically performs the following operations:

[0091] When the number of suspected merged data pairs obtained during the incremental clustering process reaches a preset number threshold, or the cumulative statistical incremental clustering completion time during the incremental clustering process reaches a preset time threshold, the suspected merged data processing flow is entered.

[0092] In a third aspect, the present invention proposes a computer device comprising a processor and a storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the incremental clustering method based on group attributes described in any of the above schemes.

[0093] In a fourth aspect, the present invention proposes a storage medium, wherein the storage medium is suitable for storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the incremental clustering method based on group attributes described in any of the above schemes.

[0094] When adopting the above-mentioned technical scheme, the present invention manually determines whether the suspected merged data pairs generated in the incremental clustering process need to be merged or split, so that erroneous mergers are suppressed and correct mergers are maintained; and in the incremental clustering processing process and when the suspected merged data pairs are manually confirmed, group attribute information and related checks are added to the clustering data, thereby enhancing the timeliness and influence of the manual intervention confirmation information, making it persistent, achieving the purpose of reinforcement learning, and solving the problem of how to effectively handle suspected merged data pairs in incremental clustering, thereby improving the accuracy and efficiency of image classification and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings, in which:

[0096] Figure 1 It is a flow chart of the main steps of the general incremental clustering method.

[0097] Figure 2 It is a flow chart of main steps of the incremental clustering method based on group attributes according to an embodiment of the present invention.

[0098] Figure 3 yes Figure 2 Specific implementation flow chart of step S206 in .

[0099] Figure 4 yes Figure 3 Flowchart of the main steps of obtaining the third clustering result according to the category discrimination result in step S2063.

[0100] Figure 5 It is a schematic diagram of the composition structure of the incremental clustering device based on group attributes of the present invention. DETAILED DESCRIPTION

[0101] Read First Figure 1 , Figure 1 This is a flow chart of the main steps of the general incremental clustering method. Figure 1 As shown, the general incremental clustering method includes:

[0102] Step S101: Obtaining data to be processed, where the data to be processed includes newly added data and historical data;

[0103] Step S102: performing incremental clustering on the data to be processed to obtain incremental clustering results;

[0104] Step S103: Update historical data according to the incremental clustering results.

[0105] In the general incremental clustering method, usually only step S101, step S102 and step S103 are included. After completing step S103, the incremental clustering process will return to step S101, and use the updated historical data and the newly added data to repeat step S101, step S102 and step S103 to complete the incremental clustering. In addition, in the general incremental clustering method, too much attention is often not paid to the processing of suspected merged data between the confirmed merge and the confirmed unmerged data, but the suspected merged data is simply included in the determined merged result, which sometimes causes clustering merge errors. In particular, with the increase in data scale and the increase in the difficulty of data distribution, the incremental clustering results are prone to superposition of merge errors, as well as more and / or large-scale disordered clustering, thereby affecting the overall clustering performance.

[0106] The incremental clustering method based on group attributes of the present invention is proposed to address the deficiency of general incremental clustering methods that they always fail to effectively handle suspected merged data. It is hoped that by introducing a method for manually identifying suspected merged data, the deficiency of the incremental clustering algorithm can be compensated, and the problem of incorrect merging of clustering results in incremental clustering can be solved, thereby improving the accuracy of clustering and making the manually confirmed information persistent, thereby achieving the purpose of reinforcement learning.

[0107] Continue reading Figure 2 , Figure 2 FIG. 1 is a flow chart of the main steps of the incremental clustering method based on group attributes according to an embodiment of the present invention. Figure 2 As shown, the incremental clustering method based on group attributes of the present invention includes:

[0108] Step S201: obtaining data to be processed, where the data to be processed includes newly added data and first clustered data;

[0109] Step S202: performing incremental clustering on the data to be processed according to the group attribute constraint to obtain an incremental clustering result, where the incremental clustering result includes the second clustering data and / or the suspected merged data;

[0110] Step S203: managing the candidate merging queue for the suspected merged data to obtain a processing priority queue for the suspected merged data pairs;

[0111] Step S204: updating the first clustering data according to the second clustering data;

[0112] Step S205: Select to execute step S201 to start a new incremental clustering, or proceed to step S206 to manually intervene in processing suspected merged data;

[0113] Step S206: Manual intervention to process suspected merged data.

[0114] In step S201, the data to be processed may be image data, such as using images of surveillance videos to perform face clustering; or it may be text data, such as using user consumption data to perform customer clustering analysis.

[0115] Each sample of the data to be processed contains label information and group attribute information. The label information is used to identify the classification result of incremental clustering, and the group attribute information is additional incremental clustering information.

[0116] The group attribute consists of a group number and a team number, and the incremental clustering process of the present invention follows the group attribute constraint. The group attribute constraint method specifically includes:

[0117] The sample with group number 0 is an invalid group attribute and has no constraint relationship with any other clustering data;

[0118] Samples whose group number is not 0 have valid group attributes;

[0119] Samples with the same group number and team number have the same label;

[0120] Samples with the same group number but different team numbers have different labels;

[0121] There is no constraint relationship between samples with different group numbers.

[0122] The first clustering data is the historical data that has completed clustering. The samples of the first clustering data contain the labels added after the incremental clustering is completed. For the newly added data, the default value of the label can be set, such as setting the default value of the label of the newly added data to 0 or other values, and the group number and team number in the initial group attributes of all samples are 0.

[0123] In addition, for the incremental data, batch (batch data) segmentation needs to be performed according to business requirements and / or resource conditions to obtain the newly added data in step S201. The method of batch segmentation is not limited in the present invention. As an example, it can be segmented according to time periods, such as setting 5 minutes as a time period. Those skilled in the art can select a batch segmentation method according to actual conditions.

[0124] In step S202, the specific algorithm of incremental clustering is not limited in the present invention. As an example, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) density clustering algorithm, hierarchical clustering algorithm, etc. can be used. Those skilled in the art can select the incremental clustering algorithm according to actual conditions.

[0125] The incremental clustering result includes the second clustering data and / or the suspected merged data. The second clustering data is the clustering data after the incremental clustering is completed and the labels are assigned.

[0126] The suspected merged data includes one or more suspected merged data pairs. In this embodiment, preferably, the suspected merged data pair includes two merged objects and the similarity score between the two merged objects, and each merged object includes its own label and at least one sample. As an example, the suspected merged data pair can be expressed as: {(labelA, sampleA1), (labelB, sampleB1), Score}, which specifically means that merged object 1 is label labelA and sample sampleA1 in label labelA, merged object 2 is label labelB and sample sampleB1 in label labelB, and the similarity score between merged object 1 and merged object 2 is Score. In addition, the merged object in the suspected merged data pair may also include multiple samples. For example, the merged object is (labelA, sampleA1, sampleA2, sampleA3), and the merged object label labelA contains 3 samples, sampleA1, sampleA2 and sampleA3.

[0127] At this time, the cluster labels labelA and labelB in the suspected merged data pair can be newly generated labels by incremental clustering, or existing labels in the first clustered data. Therefore, the combination of labels in the suspected merged data pair may be one of: new label + new label, new label + existing label, or existing label + existing label combination.

[0128] The present invention does not limit the calculation method of the similarity score. As an example, the calculation can be performed using the inner integral fraction of two merged objects. Those skilled in the art can select the calculation method of the similarity score according to actual conditions.

[0129] It should be noted that the solution of using two merged objects as a group for suspected merged data pairs is based on the consideration that the number of suspected merged data pairs is 2, which is the smallest unit, and the logical judgment is the simplest. At this time, manual judgment only needs to give a merge or non-merge result, which is very convenient for manual processing. When there are multiple merged objects in the incremental clustering results, multiple data can be paired in pairs so that each suspected merged data pair contains only two merged objects.

[0130] It should be noted that the group attributes of all samples are given by the results of manual intervention confirmation, that is, only in the relevant operation of step S206 can the group number and team number in the sample group attribute be assigned non-zero values, while the group attribute is unchanged in the incremental clustering of step S202. It is through the group attributes in the samples that the manual intervention confirmation information can be persistent and strengthen the learning of the incremental clustering process.

[0131] The group attribute constraints must be followed in the incremental clustering of step S202. For example, if the group numbers in sample sampleA1 (groupC, teamA) and sample sampleB1 (groupC, teamB) are the same, i.e., groupC, but the team numbers are different, i.e., teamA and teamB, the same label will not be assigned to sampleA1 and sampleB1 during incremental clustering.

[0132] In step S203, after each incremental clustering is completed, when the incremental clustering result contains suspected merged data, the suspected merged data pairs are updated in real time; and according to the similarity scores contained in the suspected merged data pairs, the suspected merged data pairs are sorted to form a processing priority queue. Generally, the suspected merged data pairs have a high similarity score and a high processing priority.

[0133] In step S204, the first clustering data is updated according to the second clustering data obtained in step S202.

[0134] In this embodiment, a manual intervention processing link for suspected merged data is introduced. In step S205, according to actual needs, you can choose to execute step S201 to start a new incremental clustering, or enter step S206 to manually intervene in the processing of suspected merged data. As an example, the execution condition of manual intervention can be preset, and the execution condition can be randomly preset by the user, or set according to some indicators, such as setting a threshold for the number of suspected merged data pairs, and when the set threshold is reached, reminding manual intervention; or setting an execution time threshold, starting with a certain incremental clustering as the timing starting point, and accumulating the time for each subsequent incremental clustering operation to be completed. If the accumulated time reaches the set execution time threshold, the step of manual intervention processing of suspected merged data is entered.

[0135] Continue reading Figure 3 , Figure 3 This is a specific implementation method of step S206, which includes:

[0136] Step S2061: suspending the incremental clustering operation;

[0137] Step S2062: Select the suspected merged data pair with the highest priority in the processing priority queue, perform manual discrimination, and obtain a category discrimination result;

[0138] Step S2063: Obtain third clustering data according to the category discrimination result and in accordance with the group attribute constraint;

[0139] Step S2064: updating the first clustering data according to the third clustering data;

[0140] Step S2065: Resume the incremental clustering operation and return to step S201 to start a new incremental clustering.

[0141] Considering that if manual intervention and incremental clustering are performed independently of each other, manual intervention cannot guarantee the real-time data processing. During the manual intervention judgment process, multiple incremental clustering may have been completed, and the labels of suspected merged data pairs have changed, making it impossible to guarantee the accuracy of processing priority queue data. At the same time, in order to avoid the influence of manual judgment results on incremental clustering, it is usually necessary to suspend the incremental clustering operation after executing step S2061 before starting manual judgment of suspected merged data pairs.

[0142] Before pausing incremental clustering, it is necessary to check whether there is any incremental clustering in progress; if not, directly execute step S2062; if so, wait for the ongoing incremental clustering to be completed, and execute steps S203 and S204, and then suspend the incremental clustering process to ensure that the first clustering data and the processing priority queue are the latest data.

[0143] The suspected merged data pairs with the highest priority in the processing priority queue are selected for manual discrimination to obtain category discrimination results, which include impure categories, unmerged categories, or merged categories.

[0144] In step S2063, third clustering data is obtained according to the category discrimination result, and the third clustering result is an incremental clustering result completed by manual intervention discrimination.

[0145] Continue reading Figure 4 , Figure 4 for Figure 3 Step S2063 is a flowchart of obtaining the third clustering result according to the category discrimination result.

[0146] Step S401 is executed to check the category discrimination result. First, in step S402, it is determined whether the category discrimination result is an impure category.

[0147] When the category discrimination result is an impure category, step S403 is executed to split the group attributes, and then the process goes to step S412 to obtain the third clustering result. The steps of splitting the group attributes are:

[0148] Classify the samples in the suspected merged data pairs with the same group number and team number into the first split class;

[0149] Classify the remaining samples in the suspected merged data pairs into the second split class;

[0150] Set new labels for the first split class and the second split class respectively.

[0151] As an example, the suspected merged data pair is labelA{sampleA1(groupA, teamA), sampleA2(groupA, teamA), sampleA3(groupA3, teamA3)}, labelB{sampleB1(groupB, teamB)}, label labelA contains 3 samples sampleA1, sampleA2 and sampleA3, label labelB contains 1 sample sampleB1, the group number and team number of samples sampleA1 and sampleA2 are the same, but the group number and team number of sample sampleA3 are different; after group attribute splitting, the first split class is sampleA1 and sampleA2, and the second split class is sampleA3 and sampleB1; after setting new labels for the first split class and the second split class respectively, we obtain labelC{sampleA1(groupA, teamA), sampleA2(groupA, teamA)} and labelD{sampleB1(groupB, teamB, sampleA3(groupA3, teamA3)}.

[0152] When the category discrimination result is not an impure category, steps S404 and S405 are executed to obtain the label of the merged object in the suspected merged data pair with the highest priority after manual discrimination and the sample corresponding to the label. In step S406, it is determined whether the category discrimination result is a merged category.

[0153] When the category discrimination result is not to merge the categories, step S407 is executed to assign the first set of attributes, and then the process goes to step S412 to obtain the third clustering result. The first set of attribute assignment steps are:

[0154] Assign new group numbers with the same value to the samples in the suspected merged data pairs;

[0155] Assign team numbers with different values ​​to samples corresponding to the two types of labels in the suspected merged data pair.

[0156] As an example, the suspected merged data pair is labelA{sampleA1(groupA, teamA),}, labelB{sampleB1(groupB, teamB)}; the data after the first set of attribute assignments is labelA{sampleA1(groupC, teamA1),}, labelB{sampleB1(groupC, teamB1)}. Among them, groupC is a new group number with the same value, and teamA1 and teamB1 are team numbers with different values.

[0157] When the category determination result is a merged category, step S408 is executed to perform a group attribute constraint check, and step S409 is performed to determine whether there is a conflict in the group attribute constraint check. The steps of the group attribute constraint check are:

[0158] Select all samples whose group number is not 0;

[0159] When the group number is the same and the team number is the same, the group attributes do not conflict;

[0160] When the group numbers are the same but the team numbers are different, the group attributes conflict.

[0161] When the category discrimination result is a merged category and there is a conflict in the group attribute constraint check, there is no need to process the group attributes of the sample, and the process directly proceeds to step S412 to obtain the third clustering result.

[0162] When the category discrimination result is a merged category and there is no conflict in the group attribute constraint check, step S410 and step S411 are executed, and the process proceeds to step S412 to obtain a third clustering result.

[0163] In step S410, a second set of attribute values ​​are assigned to the group attributes of the samples in the suspected merged data pair; wherein the second set of attribute values ​​are assigned as follows:

[0164] Assign new group numbers with the same value to samples of suspected merged data pairs;

[0165] Assign team numbers with the same value to samples of suspected merged data pairs.

[0166] As an example, the suspected merged data pair is labelA{sampleA1(groupA, teamA),}, labelB{sampleB1(groupB, teamB)}; the data after the second set of attribute assignments is labelA{sampleA1(groupC, teamD),}, labelB{sampleB1(groupC, teamD)}, where groupC is a new group number with the same value, and teamD is a team number with the same value.

[0167] In step S411, the suspected merged data pairs are merged to obtain a merged label and a plurality of samples corresponding to the label, and each sample has the same group number and team number.

[0168] As an example, after step S410, the suspected merged data pair is labelA{sampleA1(groupC, teamD),}, labelB{sampleB1(groupC, teamD)}. If the merge is manually judged to be merged into labelA, the merged result is labelA{sampleA1(groupC, teamD), sampleB1(groupC, teamD)}.

[0169] After the third clustering data is obtained, step S2064 is executed to update the first clustering data.

[0170] In step S204 and step S2064, the method for updating the first clustering data includes:

[0171] Add labels to the newly added data and add them to the first clustering data, for example, the second clustering data obtained after the newly added data is clustered during the normal incremental clustering process;

[0172] Modify the labels of the first clustering data. For example, after the suspected merged data pair is processed by manual intervention, the labels are merged. It may be necessary to modify the labels of some samples in the first clustering data according to the third clustering data.

[0173] In step S2065, the incremental clustering operation is resumed, and new incremental clustering is started from step S201.

[0174] It should be noted that after executing step S2064, you can choose to execute step S2065 to start a new incremental clustering; you can also choose to return to step S2062 and select the suspected merged data pair with the highest priority in the remaining processing priority queue to continue manual identification. Therefore, according to the actual situation, you can choose the number of suspected merged data pairs to be processed each time by manual intervention, and you can process only one at a time, or you can process multiple pairs continuously each time.

[0175] Furthermore, the present invention also provides an incremental clustering device based on group attributes. Figure 5 As shown, the incremental clustering device 5 based on group attributes of the present invention mainly includes: a data loading module 51, an incremental clustering module 52, a suspected merged data management module 53, a human-computer interaction determination module 54 and a historical data maintenance module 55.

[0176] As an example, the data loading module 51 is configured to execute step S201 to obtain the data to be processed. The incremental clustering module 52 is configured to execute the operation in step S202. The suspected merged data management module 53 is configured to execute the operation in step S203. The human-computer interaction discrimination module 54 is configured to execute the operations in steps S205, S206, S2061, S2062, S2063, S2065 and S401 to S412. The historical data maintenance module 55 is configured to execute the operations in steps S204 and S2064.

[0177] Furthermore, the present invention also provides a computer device, which includes a processor and a storage device, the storage device can be configured to store and execute the program of the incremental clustering method based on group attributes of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the incremental clustering method based on group attributes of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The incremental clustering device based on group attributes can be a control device device formed by various electronic devices.

[0178] Furthermore, the present invention also provides a storage medium, which can be configured to store a program for executing the incremental clustering method based on group attributes of the above-mentioned method embodiment, and the program can be loaded and run by a processor to implement the above-mentioned incremental clustering method based on group attributes. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The storage medium can be a storage device formed by various electronic devices. Optionally, the storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0179] Those skilled in the art should be able to appreciate that the method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0180] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present invention and the above-mentioned drawings are only used to distinguish similar objects, rather than to describe or indicate a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0181] It should be noted that, in the description of this application, the term "A and / or B" represents all possible combinations of A and B, such as only A, only B, or A and B.

[0182] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. An incremental clustering method based on group attributes, characterized in that: The method comprises: S1, obtaining data to be processed, wherein the data to be processed includes newly added data and first clustering data, wherein the first clustering data is historical data for which clustering has been completed, and the data type of the data to be processed includes image data or text data, and each sample in the data to be processed includes group attributes, and the group attributes include a group number and a team number; S2, performing incremental clustering on the data to be processed in accordance with the group attribute constraint to obtain a clustering result, wherein the clustering result includes the second clustering data and / or the suspected merged data that have completed the incremental clustering, wherein the suspected merged data includes one or more suspected merged data pairs, wherein the suspected merged data pairs include two merged objects and a similarity score between the two merged objects, and each merged object includes a label and at least one sample; S3, updating the first clustering data according to the second clustering data; S4, performing candidate merging queue management on the suspected merged data to obtain a processing priority queue of the suspected merged data pair; S5, in response to reaching the preset execution condition, entering the suspected merged data processing flow, otherwise continuing to execute step S1; wherein the preset execution condition includes that the number of the suspected merged data pairs obtained in step S2 reaches a preset number threshold, or the cumulative statistical incremental clustering completion time in step S2 reaches a preset time threshold; The steps of the "suspected merged data processing flow" specifically include: Pause incremental clustering operations; Obtaining the suspected merged data pair with the highest priority in the processing priority queue, and manually distinguishing the suspected merged data pair with the highest priority to obtain a category distinction result, wherein the category distinction result includes one of an impure category, an unmerged category, or a merged category; According to the category discrimination result and in compliance with the group attribute constraint, third clustering data is obtained; Updating the first clustering data according to the third clustering data; Resume the incremental clustering operation and execute step S1 to start new incremental clustering.

2. The incremental clustering method based on group attributes according to claim 1, characterized in that: The step of "obtaining third clustering data according to the category discrimination result and following the group attribute constraint" specifically includes: When the category discrimination result is an impure category, performing group attribute splitting on the samples of the suspected merged data pair; When the category discrimination result is a non-merging category, assigning a first group of attributes to the group attributes of the sample of the suspected merged data pair; When the discrimination result is a merge category, a group attribute constraint check is performed on the samples of the suspected merged data pair, and it is determined whether to merge the suspected merged data pair according to the result of the group attribute constraint check.

3. The incremental clustering method based on group attributes according to claim 2, characterized in that: The step of "when the category discrimination result is an impure category, performing group attribute splitting on the samples of the suspected merged data pair" specifically includes: Classify the samples in the suspected merged data pairs with the same group number and team number into a first split class; Classify the remaining samples in the suspected merged data pairs into a second split class; New labels are set for the first split class and the second split class respectively.

4. The incremental clustering method based on group attributes according to claim 2, characterized in that: The step of "when the category discrimination result is a non-merging category, assigning a first group of attributes to the group attributes of the sample of the suspected merged data pair" specifically includes: Assigning a new group number with the same value to the samples in the suspected merged data pair; Team numbers with different values ​​are respectively assigned to samples corresponding to the two types of labels in the suspected merged data pair.

5. The incremental clustering method based on group attributes according to claim 2, characterized in that: The step of "when the discrimination result is a merge category, performing a group attribute constraint check on the samples of the suspected merged data pair, and determining whether to merge the suspected merged data pair according to the result of the group attribute constraint check" specifically includes: Checking whether the group attributes of all samples in the suspected merged data pair conflict; If there is a conflict, it will not be merged; If there is no conflict, assigning a second group of attributes to the group attributes of the samples in the suspected merged data pair; According to the second set of attribute values, merging the suspected merged data pairs; The step of "assigning a second group of attributes to the group attributes of the samples in the suspected merged data pair" specifically includes: Allocating a new group number with the same value to the samples of the suspected merged data pair; The team numbers with the same values ​​are assigned to the samples of the suspected merged data pairs.

6. The incremental clustering method based on group attributes according to claim 5, characterized in that: The step of "checking whether the group attributes of all samples in the suspected merged data pair conflict" specifically includes: Select all samples whose group number is not 0; When the group numbers are the same and the team numbers are the same, the group attributes do not conflict; When the group numbers are the same but the team numbers are different, the group attributes conflict.

7. The incremental clustering method based on group attributes according to any one of claims 1 to 6, characterized in that: The method of group attribute constraints includes: The sample with group number 0 is an invalid group attribute and has no constraint relationship with any other clustering data; The samples whose group number is not 0 have a valid group attribute; The samples with the same group number and team number have the same label; The samples with the same group number but different team numbers have different labels; There is no constraint relationship between the samples with different group numbers.

8. An incremental clustering device based on group attributes, characterized in that: The device comprises: A data loading module, wherein the data acquisition module is configured to acquire data to be processed, wherein the data to be processed includes newly added data and first clustering data, wherein the first clustering data is historical data for which clustering has been completed, and the data type of the data to be processed includes image data or text data, and each sample in the data to be processed includes group attributes, wherein the group attributes include a group number and a team number; An incremental clustering module, wherein the incremental clustering module is configured to perform incremental clustering on the data to be processed in accordance with the group attribute constraint to obtain a clustering result, wherein the clustering result includes the second clustering data for which incremental clustering has been completed and the suspected merged data, wherein the suspected merged data includes one or more suspected merged data pairs, wherein the suspected merged data pairs include two merged objects and a similarity score between the two merged objects, and each merged object includes a label and at least one sample; a historical data maintenance module, wherein the historical data maintenance module is configured to update the first clustering data according to the second clustering data; a suspected merged data management module, wherein the suspected merged data management module is configured to perform candidate merge queue management on the suspected merged data to obtain a processing priority queue of the suspected merged data pairs; A human-computer interaction discrimination module, wherein the human-computer interaction discrimination module is configured to, in response to reaching a preset execution condition, enter a suspected merged data processing flow, otherwise continue to execute the incremental clustering process, wherein the preset execution condition includes that the number of the suspected merged data pairs obtained in the incremental clustering process reaches a preset number threshold, or the cumulative statistical incremental clustering completion time in the incremental clustering process reaches a preset time threshold; The human-computer interaction determination module is further configured to perform the following operations: Pause incremental clustering operations; Obtaining the suspected merged data pair with the highest priority in the processing priority queue, and manually distinguishing the suspected merged data pair with the highest priority to obtain a category distinction result, wherein the category distinction result includes one of an impure category, an unmerged category, or a merged category; According to the category discrimination result and in compliance with the group attribute constraint, third clustering data is obtained; Updating the first clustering data according to the third clustering data; By means of a historical data maintenance module, the first clustering data is updated according to the third clustering data; Resume the incremental clustering process and start a new incremental clustering.

9. The incremental clustering device based on group attributes according to claim 8, characterized in that: The human-computer interaction determination module is further configured to perform the following operations: When the category discrimination result is that the category is an impure category, performing group attribute splitting on the samples of the suspected merged data pair; When the category discrimination result is a non-merging category, assigning a first group of attributes to the group attributes of the sample of the suspected merged data pair; When the discrimination result is a merge category, a group attribute constraint check is performed on the samples of the suspected merged data pair, and it is determined whether to merge the suspected merged data pair according to the result of the group attribute constraint check.

10. The incremental clustering device based on group attributes according to claim 9, characterized in that: The human-computer interaction identification module specifically performs the following operations: Classify the samples in the suspected merged data pairs with the same group number and team number into a first split class; Classify the remaining samples in the suspected merged data pairs into a second split class; New labels are set for the first split class and the second split class respectively.

11. The incremental clustering device based on group attributes according to claim 9, characterized in that: The human-computer interaction identification module specifically performs the following operations: Assigning a new group number with the same value to the samples in the suspected merged data pair; Team numbers with different values ​​are respectively assigned to samples corresponding to the two types of labels in the suspected merged data pair.

12. The incremental clustering device based on group attributes according to claim 9, characterized in that: The human-computer interaction identification module specifically performs the following operations: Checking whether the group attributes of all samples in the suspected merged data pair conflict; If there is a conflict, it will not be merged; If there is no conflict, assigning a second group of attributes to the group attributes of the samples in the suspected merged data pair; According to the second set of attribute values, merging the suspected merged data pairs; The step of "assigning a second group of attributes to the group attributes of the samples in the suspected merged data pair" specifically includes: Allocating a new group number with the same value to the samples of the suspected merged data pair; The team numbers with the same values ​​are assigned to the samples of the suspected merged data pairs.

13. The incremental clustering device based on group attributes according to claim 12, characterized in that: The human-computer interaction identification module specifically performs the following operations: Select all samples whose group number is not 0; When the group numbers are the same and the team numbers are the same, the group attributes do not conflict; When the group numbers are the same but the team numbers are different, the group attributes conflict.

14. The incremental clustering device based on group attributes according to any one of claims 8 to 13, characterized in that: The method of group attribute constraints includes: The sample with group number 0 is an invalid group attribute and has no constraint relationship with any other clustering data; The samples whose group number is not 0 have a valid group attribute; The samples with the same group number and team number have the same label; The samples with the same group number but different team numbers have different labels; There is no constraint relationship between the samples with different group numbers.

15. A computer device comprising a processor and a storage device, wherein the storage device is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and executed by the processor to execute the incremental clustering method based on group attributes according to any one of claims 1 to 7.

16. A storage medium, the storage medium being suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the incremental clustering method based on group attributes according to any one of claims 1 to 7.

Images