Incremental clustering method and device based on flow mode

Abstract

The embodiment of the present invention discloses an incremental clustering method and device based on streaming modality, the method comprising: obtaining new modality information at the current moment of multiple view data samples and historical inheritance comprehensive information of all modalities before the current moment; for any sample, performing similarity decomposition and discriminant learning on the new modality information and the historical inheritance comprehensive information to obtain the overall objective function; solving the similarity decomposition result of the new modality information and the historical inheritance comprehensive information that satisfies the overall objective function, and calculating the historical inheritance comprehensive information including the current moment based on the similarity decomposition result; clustering multiple samples based on the historical inheritance comprehensive information including the current moment of each sample to obtain clustering results. Through the above manner, the embodiment of the present invention studies the clustering problem of streaming modality scenarios for the first time, and can accurately and effectively perform incremental clustering on streaming modality samples.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of stream modality data processing, and in particular to a stream modality-based incremental clustering method and device. Background Art

[0002] In this era of information explosion, people are overwhelmed by massive amounts of data. As the data we face becomes increasingly diverse and rich, it is difficult to meet people's needs for comprehensive understanding and analysis of data by using only single-modal information. Therefore, multimodal / multi-view learning methods have emerged. By fusing information from different views, the diversity of data can be more fully utilized to improve the robustness and performance of the model. However, in practical tasks, it is difficult to obtain a large number of labeled samples due to the difficulty and high cost of obtaining data labels. This brings higher challenges and requirements to multimodal learning. In this case, multimodal clustering becomes a powerful tool. It does not rely on the label information of the data and can process unlabeled data more flexibly. Therefore, multimodal clustering has become one of the hot topics in academic research and is widely used in machine learning, image processing and other fields.

[0003] In previous studies, it is important to note that most of the positive results obtained through multimodal clustering depend on a basic assumption that all modalities are static and can be accessed simultaneously. However, in many practical applications, especially in the digital process where the environment is constantly changing and the speed of data generation and transmission is accelerating, such an assumption is difficult to hold. Take the analysis of personal interests and hobbies through social platforms as an example. With the popularity of social media, people express their interests and hobbies by sharing life fragments such as audio, photos, text or video. If people are regarded as sample points and messages posted on the platform are regarded as modalities, then the modalities will gradually increase over time if the number of samples remains unchanged. However, existing social platforms can limit the visibility of shared content, such as only allowing viewing within a certain time range (such as one or three days). In traditional methods, all modes need to be obtained in advance, which means that all historical modes must be saved once they appear. Therefore, the number of modalities used for analysis will continue to accumulate over time, and if the data is stored and the model is restarted every time a modality is reached, it will consume a lot of storage resources and lead to intensive computing costs. As Figure 1 As shown in Figure 1, in this scenario, the number of samples remains unchanged, the number of modalities is uncertain and continues to increase over time, and is called a streaming modality. However, traditional methods require us to save and collect published content every time, and then combine all modalities for analysis and update. Therefore, it is obvious that traditional multimodal clustering methods are not suitable for dynamic multimodal clustering scenarios.

[0004] To solve this problem, a direct approach is to adjust the traditional clustering method. On the one hand, the traditional method requires processing all modalities at the same time. By analogy, it means that every time a new modality appears, all the modalities that have appeared must be accessed, which will lead to redundant calculations and large storage requirements. On the other hand, the traditional method is mainly designed for static scenes and lacks the ability to effectively handle dynamic scenes. So far, supervised solutions can be used. It can alleviate the expensive computational cost of repeatedly retraining the model when processing streaming modalities to a certain extent, and achieve the purpose of real-time updating. However, it requires a large number of modalities for initialization, and all the original data needs to be continuously saved for fine-tuning of the model. When the number and dimension of modalities are large, this still requires a lot of storage costs. Therefore, the algorithm does not fundamentally solve the problem of real-time inheritance and updating of streaming modalities. Summary of the invention

[0005] In view of the above problems, an embodiment of the present invention provides a flow modality-based incremental clustering method and device, which overcomes the above problems or at least partially solves the above problems.

[0006] According to one aspect of an embodiment of the present invention, there is provided an incremental clustering method based on streaming modality, the method comprising: obtaining new modality information at a current moment of multiple view data samples and historical inheritance comprehensive information of all modalities before the current moment; for any sample, performing similarity decomposition and discriminant learning on the new modality information and the historical inheritance comprehensive information to obtain a total objective function; solving a similarity decomposition result of the new modality information and the historical inheritance comprehensive information that satisfies the total objective function, and calculating the historical inheritance comprehensive information including the current moment based on the similarity decomposition result; clustering multiple samples based on the historical inheritance comprehensive information including the current moment of each sample to obtain a clustering result.

[0007] In an optional manner, the similarity decomposition and discriminant learning are performed on the new modal information and the historically inherited comprehensive information to obtain an overall objective function, including: performing similarity decomposition on the new modal information and the historically inherited comprehensive information to obtain a first objective function; performing discriminant learning on the new modal information and the historically inherited comprehensive information between modalities and within modalities to obtain a second objective function; and combining the first objective function and the second objective function to obtain an overall objective function.

[0008] In an optional manner, the new modal information and the historical inherited comprehensive information are subjected to similarity decomposition to obtain a first objective function, including: decomposing the new modal information into a consistency information matrix of all modalities before the current moment and specific information of the new modality; decomposing the historical inherited comprehensive information into the consistency information matrix and specific information of the historical modality; and obtaining the first objective function of similarity decomposition based on the consistency information matrix, the specific information of the new modality and the specific information of the historical modality combined with the real-time update characteristics of streaming modal data.

[0009] In an optional manner, the new modal information and the historical inherited comprehensive information are discriminatively learned between modalities and within modalities to obtain a second objective function, including: obtaining a first judgment result based on the different information carried by different modalities, the first judgment result being that there is a significant difference between the specific information of the new modality and the specific information of the historical modality; obtaining a second judgment result based on that different attributes of the internal information of a single modality have different meanings, the second judgment result being that there is a significant difference between the similarity matrix and the specific information of the new modality and the specific information of the historical modality; and obtaining a second objective function of discriminative learning based on the first judgment result and the second judgment result.

[0010] Optionally, the combining of the first objective function and the second objective function to obtain the total objective function includes: solving the minimized trace of the Laplace matrix corresponding to the consistency information according to the flow modal clustering property; and obtaining the total objective function according to the first objective function, the second objective function and the minimized trace of the Laplace matrix.

[0011] Optionally, the overall objective function is:

[0012]

[0013] Among them, A T is the consistency information, λ, β, γ are weight parameters, α t is the coefficient at time t, and α is the T and α T-1 The relevant vector, S T is the new modal information at time T, P T is the specific information of the new modality at time T, S T-1 is the historical modal information before time T, P T-1 is the specific information of the historical mode before time T, L A is the Laplace matrix corresponding to the consistency information, and F is the label matrix.

[0014] Optionally, the calculating the historical inheritance comprehensive information including the current moment according to the similarity decomposition result includes: calculating the historical inheritance comprehensive information including the current moment according to the similarity decomposition result by applying the following relational expression:

[0015]

[0016] in, A is the historical inheritance comprehensive information including the current moment. T is the consistency information, P T is the specific information of the new modality at time T, P T-1 It is the specific information of the historical modality before the current time T.

[0017] Based on the same inventive concept, a stream modality-based incremental clustering device is provided, comprising: a modality information acquisition unit, used to acquire new modality information of multiple view data samples at the current moment and historical inheritance comprehensive information of all modalities before the current moment; an objective function acquisition unit, used to perform similarity decomposition and discriminant learning on the new modality information and the historical inheritance comprehensive information for any sample, to obtain a total objective function; an objective solving unit, used to solve the similarity decomposition result of the new modality information and the historical inheritance comprehensive information that satisfies the total objective function, and calculate the historical inheritance comprehensive information including the current moment according to the similarity decomposition result; a clustering unit, used to cluster multiple samples according to the historical inheritance comprehensive information including the current moment of each sample, to obtain a clustering result.

[0018] Based on the same inventive concept, an embodiment of the present invention further proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned method when executing the program.

[0019] Based on the same inventive concept, an embodiment of the present invention further proposes a computer storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to execute the aforementioned method.

[0020] The embodiment of the present invention obtains the new modal information of multiple view data samples at the current moment and the historical inheritance comprehensive information of all modalities before the current moment; for any sample, similarity decomposition and discriminant learning are performed on the new modal information and the historical inheritance comprehensive information to obtain a total objective function; the similarity decomposition result of the new modal information and the historical inheritance comprehensive information that satisfies the total objective function is solved, and the historical inheritance comprehensive information including the current moment is calculated according to the similarity decomposition result; multiple samples are clustered according to the historical inheritance comprehensive information including the current moment of each sample to obtain a clustering result, and the clustering problem of the streaming modal scenario is studied for the first time, and the streaming modal samples can be accurately and effectively incrementally clustered.

[0021] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 shows an example graph of flow modal data;

[0024] Figure 2 A schematic diagram of a flow chart of an incremental clustering method based on flow mode provided in an embodiment of the present invention is shown;

[0025] Figure 3 A schematic diagram of flow mode clustering according to an embodiment of the present invention is shown;

[0026] Figure 4 A schematic diagram of 8 multimodal data sets used for experiments provided by an embodiment of the present invention is shown;

[0027] Figure 5 A schematic diagram showing the accuracy of clustering results of different methods provided by embodiments of the present invention;

[0028] Figure 6 A schematic diagram of the F-score of clustering results of different methods provided by embodiments of the present invention is shown;

[0029] Figure 7 A schematic diagram of the Rand index of clustering results of different methods provided by embodiments of the present invention is shown;

[0030] Figure 8 A schematic diagram showing the recall rate of clustering results of different methods provided in the embodiments of the present invention is shown;

[0031] Fig. 9 A schematic diagram showing a comparison between dynamic flow modal clustering and static multimodal clustering provided by an embodiment of the present invention is shown;

[0032] Fig.10 A schematic diagram of importance evaluation of different parts of an incremental clustering method based on flow mode provided by an embodiment of the present invention is shown;

[0033] Fig.11 A schematic diagram of analyzing different parameters in the incremental clustering method based on flow mode provided by an embodiment of the present invention is shown;

[0034] Fig.12 A schematic diagram of the structure of an incremental clustering device based on flow mode provided by an embodiment of the present invention is shown;

[0035] Fig.13 A schematic diagram of an electronic device in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0036] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0037] Figure 2 FIG. 1 is a flow chart of an incremental clustering method based on flow mode provided by an embodiment of the present invention. Figure 2 As shown, the incremental clustering method based on the flow mode is applied to the server, and the incremental clustering method based on the flow mode includes:

[0038] Step S11: obtaining new modality information of multiple view data samples at the current moment and historical inheritance comprehensive information of all modalities before the current moment.

[0039] In the embodiment of the present invention, N samples consisting of V modes are given and expressed as in represents the Tth mode of the ith sample. Each data point All belong to one of C categories. The feature matrices of different modes are represented as {X1,…,X T ,…},in It is worth noting that bold uppercase and bold lowercase letters are used to represent matrices and vectors, respectively, and the identity matrix is ​​represented by I.ij and A ij Both represent matrices The (i,j)th element of A=[a1,a2,…,a N ], the l1 norm of matrix A can be expressed as

[0040] In step S11, the new mode X T After it appears, obtain the new modality information of the current time T of multiple view data samples and the historical inheritance comprehensive information of all modalities before the current time T. New modality X T That is, the new increment in the incremental clustering. The view data may be one or more of image, audio, and video data, which is not specifically limited here.

[0041] Figure 3 It is a schematic diagram of flow mode clustering. Assume that at time T-1, the historical inheritance comprehensive information for inheritance of all modes before time T (from time 1 to time T-1) is obtained. At time T the new mode X T After it appears, the corresponding similarity matrix S is obtained through similarity learning. T , and expand on this basis. From time 1 to time T, it shows that time moves forward step by step, where If the current time is time T, to avoid confusion between characters in the figure, the specific information of the historical mode is expressed as Since the subsequent analysis is based on time periods, it can be directly expressed as P T-1 .

[0042] Step S12: For any sample, similarity decomposition and discriminative learning are performed on the new modal information and the historical inherited comprehensive information to obtain the overall objective function.

[0043] The key to the streaming modal clustering method is to inherit historical information and avoid information forgetting in the absence of data labels, while ensuring the performance of the clustering algorithm. Streaming modal data has the characteristic of increasing feature dimensions. Therefore, how to inherit historical information and update the model at the same time is the key to streaming modal learning. The embodiment of the present invention uses sample similarity to characterize the inherited information.

[0044] In the embodiment of the present invention, optionally, the new modal information and the historical inherited comprehensive information are firstly subjected to similarity decomposition to obtain a first objective function. Optionally, the new modal information is decomposed into a consistency information matrix of all modalities before the current moment and specific information of the new modality; the historical inherited comprehensive information is decomposed into the consistency information matrix and specific information of the historical modality; the first objective function of similarity decomposition is obtained based on the consistency information matrix, the specific information of the new modality and the specific information of the historical modality combined with the real-time update characteristics of the streaming modal data.

[0045] In the embodiment of the present invention, various modalities gradually emerge over time. The stream modality is essentially used to describe the same research object, so finding shared information between modalities is a common processing strategy. Considering that the newly arrived modality may carry fresher and more timely information, analysis is performed from the current moment (moment T) and historical moments (time 1 to time T-1). If only the consistency information between modalities is considered, the following formula can be obtained:

[0046]

[0047] in, represents the new modal information at time T, and Represents the residual term in different time periods. T It is the consistency information of all modes before time T, and it will be continuously improved over time. T-1 Represents the historical modal information of all modes that appeared before time T and satisfies the following equation:

[0048]

[0049] It is the historical inheritance comprehensive information of all modalities before time T. The above formula indicates that the modal information is transitioned from the previous stage (T-1 moment) to the current stage (T moment). Based on this, the similarity decomposition can be expressed as:

[0050]

[0051] represents the F-normal form, Ω(A T ) represents the function value of similarity decomposition. Since each modality has its own unique characteristics, the division considered in equation (1) is not comprehensive enough. Considering that the modality appearing at different time points should contain inherent differences that are different from other modalities. Therefore, considering A comprehensively T , we get the following relationship:

[0052]

[0053] in, and Represents the specific information of the new modality at the current time T and the specific information of all modalities before the current time T.

[0054] According to the characteristics of real-time update of flow modal data, taking time T as an example, the first objective function of similarity decomposition of the following flow modal can be obtained:

[0055]

[0056] Among them, A T is the consistency information of all modes before time T, represents the 2-normal form, γ is the weight parameter, α t is the coefficient at time t, and α is the T and α T-1 Related vectors.

[0057] Then, the new modal information and the historical inherited comprehensive information are discriminated from between modalities and within modalities to obtain a second objective function. Optionally, a first judgment result is obtained based on the different information carried by different modalities, and the first judgment result is that there is a significant difference between the specific information of the new modality and the specific information of the historical modality; a second judgment result is obtained based on the different meanings of different attributes of the internal information of a single modality, and the second judgment result is that there is a significant difference between the similarity matrix and the specific information of the new modality and the specific information of the historical modality; a second objective function of discriminative learning is obtained based on the first judgment result and the second judgment result.

[0058] In the embodiment of the present invention, the discriminative learning of flow modes can be analyzed from two perspectives: inter-modality and intra-modality. On the one hand, the information carried by different modes is different, so T and P T-1 Create a significant difference between them, so that information from different stages can be distinguished. The calculation formula is as follows:

[0059] min||P T ⊙P T-1 ||1 (6)

[0060] Where ⊙ represents the Hammond product of the matrix, and (P T ⊙P T-1 ) ij =(P T ) ij (P T-1 ) ij The special definition of the Hammond product of matrices gives it a significant advantage in measuring the differences between matrices. For example, minimizing ||PT ⊙P T-1 The goal of ||1 has two meanings: first, for non-zero elements, the expected matrix P T and P T-1 The values ​​of the corresponding positions of are as different as possible; secondly, if P T and P T-1 If all elements of are 0 (i.e., trivial solution), the objective function is minimized, but there is no significant difference between them.

[0061] On the other hand, as described in equation (1), different attributes of the internal information of a single modality have different meanings, which can be further divided into the consistency information A T and specific information P T (P T-1 ). Similarly, we hope that in the same information A T and specific information P T (P T-1 ) to create as much gap as possible between them to facilitate the inheritance of core information, which can be specifically expressed as follows:

[0062]

[0063] Taking into account the above two requirements and the properties of the Handmade product, taking time T as an example, the second objective function is obtained by discriminant learning based on flow mode as follows:

[0064]

[0065] After obtaining the first objective function and the second objective function, the total objective function is obtained by combining the first objective function and the second objective function. Optionally, the trace of the Laplacian matrix corresponding to the consistency information that is minimized is solved according to the flow modal clustering property; the total objective function is obtained according to the first objective function, the second objective function and the trace of the minimized Laplacian matrix. In an embodiment of the present invention, according to the above analysis of the two parts of flow modal similarity decomposition and difference learning, A T Represents the similarity (i.e., consistency information) of all modes up to time T. Since the optimal similarity clustering structure should satisfy the condition that the number of connected branches in the similarity matrix is ​​equal to the number of clusters. This similarity structure with this characteristic can greatly help the subsequent clustering process. It is well known that the number of connected components in the similarity matrix is ​​equal to the multiplicity of the eigenvalue 0 of the corresponding Laplace matrix, that is, if the similarity matrix A T The component in is exactly equal to C, then the sample can be divided into C clusters. However, the solution A obtained above T may not satisfy the desired properties. Ideally, when the similarity matrix A T When C-connected components are included, the corresponding Laplacian matrix LA The rank of should be NC. However, to handle the rank constraint rank(L A ) = NC proves to be challenging. This problem can be solved by transforming it to minimize the trace of the matrix.

[0066] Ω(F)=tr(F T L A F) (9)

[0067] in, And F T F=I.

[0068] Thus, the total objective function is obtained according to the first objective function, the second objective function and the trace of the minimized Laplace matrix as follows:

[0069]

[0070] Among them, A T is the consistency information, λ, β, γ are weight parameters, and the selection range is {10 -3 , 10 -2 , …, 10 2 , 10 3}, α t is the coefficient at time t, and α is the T and α T-1 The relevant vector, S T is the new modal information at time T, P T is the specific information of the new modality at time T, S T-1 is the historical modal information before time T, P T-1 is the specific information of the historical mode before time T, L A is the Laplace matrix corresponding to the consistency information, and F is the label matrix.

[0071] Step S13: solving the similarity decomposition result of the new modal information and the historical inheritance comprehensive information that satisfies the overall objective function, and calculating the historical inheritance comprehensive information including the current moment according to the similarity decomposition result.

[0072] In the embodiment of the present invention, an alternating iteration strategy is applied to solve the total objective function determined in step S12 to obtain a similarity decomposition result of the new modal information and the historical inheritance comprehensive information: A T ,α,F,P T-1 , P T , according to the similarity decomposition result, the following relational expression is applied to calculate the historical inheritance comprehensive information including the current moment:

[0073]

[0074] in, A is the historical inheritance comprehensive information including the current time T, that is, the historical inheritance comprehensive information of all modalities before time T+1, T is the consistency information of all modes before the current time T, P T is the specific information of the new modality at time T, P T-1 The specific information of the historical modality before the current moment T. This includes the historical inheritance comprehensive information of the current moment T It can be directly used for spectral clustering to obtain clustering results, or used as inherited information to participate in model learning at time T+1. For inheritance information.

[0075] At time T+1, mode X T+1 appears. Save history information and rewrite it as S T , and then learn the new modality X T+1 The similarity matrix S T+1 Next, the inherited information S T With current information T+1 Combined, the steps in equation (10) are repeated. In this way, the inherited information can be continuously learned and updated.

[0076] Step S14: clustering multiple samples according to the historical inheritance comprehensive information of each sample including the current moment to obtain a clustering result.

[0077] In the embodiment of the present invention, after obtaining the historical inheritance comprehensive information including the current time T of each sample according to step S12 and step S13, multiple samples are clustered according to the historical inheritance comprehensive information including the current time T, and all samples are clustered into C categories to obtain clustering results.

[0078] The incremental clustering method based on streaming modality in the embodiment of the present invention aims at the streaming modality problem where new modalities continue to emerge over time. It updates the model by jointly analyzing the historical and current modality information, identifies different modalities through similarity decomposition, and maximizes the differences to enhance the consistency information. The obtained consistency information contains the precisely expected connected components to ensure the accuracy of clustering; based on the learned consistency information and specific information, an iteratively updated comprehensive similarity is constructed as inheritance information for clustering. By effectively inheriting historical information, the problem of information being forgotten over time is avoided. It also has real-time update capabilities and can generate new inheritance information while processing new and historical modalities. The incremental clustering method based on streaming modality in the embodiment of the present invention studies the clustering problem of streaming modality scenarios for the first time, providing new insights for streaming modality learning. Through genetic-based similarity decomposition and discriminant learning methods, the model is continuously learned and updated through effective inheritance, and the samples of streaming modality can be effectively and incrementally clustered, showing considerable application potential, and can be applied to the diagnosis of schizophrenia, etc.

[0079] The following is a performance evaluation of the incremental clustering method based on flow mode in the embodiment of the present invention. Figure 4 Experiments were conducted on the 8 multimodal datasets shown. Since incremental clustering based on streaming modality is a new scenario, a contrast method was selected when designing the experiment to perform clustering by learning a consistent similarity matrix (or similarity matrix, collectively referred to as similarity matrix). ICSM is a method based on modality similarity.

[0080] In order to demonstrate the advantages of dynamic modal learning (ICSM) of an embodiment of the present invention over static modal learning, ICSM adopts the same similarity learning strategy as the comparison method in each modality, and two types of comparison experiments are designed on this basis. On the one hand, the first comparison method is designed to illustrate the importance of information inheritance. ICSM is an incremental clustering method with dynamic inheritance, while general methods do not have inheritance. Therefore, the comparison method is set to cluster only the current modality and has no ability to inherit historical information. These methods are uniformly named with the suffix "-Sn". On the other hand, the second comparison method is further designed to illustrate the advantages of global inheritance over local inheritance. Since ICSM inherits a comprehensive similarity matrix of size N×N, a similarity matrix of size N×N is also inherited in the comparison method. The difference is that the comparison method only inherits the similarity matrix of the previous modality, rather than the global similarity matrix of all modalities that have appeared in history. The specific inheritance strategy is to inherit the similarity matrix information S of the previous modality. T-1 Similarity matrix information S with the current modality T Combined, the mean is taken before clustering, that is, 0.5×(S T-1+S T ). These methods are named with the suffix "-Sc".

[0081] The specific comparison methods are as follows: (1) CLR (Constrained Laplace Rank Algorithm for Graph-based Clustering), which is a single-view similarity learning method; (2) MVGC (Sample-level Multi-view Graph Clustering), which is quite stable under different parameter settings; (3) JMVFG (Joint Multi-view Unsupervised Feature Selection and Graph Learning), which is consistent and robust under different parameter settings; (4) GMC (Graph-based Multi-view Clustering), the parameters of GMC can be automatically tuned. Therefore, based on the above four different similarity calculation methods, four variants of the incremental clustering method based on streaming modality are obtained, namely ICSM1-ICSM4. It is worth noting that the multimodal clustering model is not started at each stage because this method is not suitable for the scenario of dynamic clustering with streaming modalities and does not store any modal data information. In addition, as the number of modalities continues to increase, the computational load becomes too high, making this algorithm impractical in real-world applications.

[0082] The clustering results of 12 methods on 8 data sets are shown in Figure 5-8 . The arrows in each row represent the streaming media mode over time. For example, 1→5 represents the progression from the first mode to the second, third, fourth and finally fifth mode. The clustering performance is evaluated by accuracy (ACC), F-score, Rand Index (RandIndex) and recall (Recall). The higher the score, the better the result, and the best result is highlighted in bold in the figure. It can be seen from the figure that when the modality is presented in a streaming manner, the dynamic incremental clustering method of the embodiment of the present invention is significantly better than the static modal clustering and short-term modal average clustering methods in most cases. And with the addition of more modalities, the advantages of the method of the embodiment of the present invention gradually emerge, which fully demonstrates the advantages of ICSM in the streaming modality.

[0083] First, the incremental clustering method based on streaming modality inherits the comprehensive similarity information of historical data, avoids the forgetting of historical information, reduces the continuous storage of modalities, and considers the impact of historical modal information on current clustering. Therefore, in view of the challenge of large-scale data processing faced by streaming learning, choosing appropriate inherited information can effectively avoid catastrophic forgetting. Second, according to the results of the "-Sn" method, it can be seen that only considering the modal information currently arriving, although good results are sometimes achieved, it is difficult to maintain consistent good performance. For example, in the Handwritten dataset, CLR-Sn achieved the best performance in terms of accuracy using the CLR method in stages 1→3 and 1→5. However, this may be a chance occurrence, which depends entirely on whether the information contained in the currently reached mode is comprehensive. Therefore, the results of the "-Sn" method may fluctuate greatly. Similarly, the "-Sc" method based on the modal information of the current and previous stages obtains better and more stable results than "-Sn". But in fact, it ignores some historical modal information, especially when the time becomes longer and involves more diverse modalities. In this case, the difference between the "-Sc" method and the ICSM method that fully considers historical information becomes more obvious. Finally, regardless of the number of modes experienced, the incremental clustering method based on flow modality of the embodiment of the present invention has achieved very promising results and has always been in a relatively stable state over time. Since the amount of information carried by each modality may be different at different times, it may be completely new information, or it may have some overlap with the information carried by historical modalities. It is normal for the ICSM results to show fluctuations. In summary, the ICSM method effectively solves the problem of flow modality and achieves competitive clustering results. It also shows a certain degree of robustness when faced with changes in evaluation indicators.

[0084] Let’s continue to discuss the difference between using streaming mode (ICSM) and static multimodal clustering without considering storage cost. For example, given the dataset Caltech1010-7, which contains 5 modalities in total, the dynamic streaming modal clustering will access these 5 modalities in sequence, while the static multimodal clustering will access these 5 modalities at the same time. Fig. 9 The results on Caltech101-7 and Yale datasets are shown in Figure 2. ICSM2-ICSM4 represent streaming modal clustering methods; MVGC, JMVFG, and GMC represent multimodal clustering methods, and show the clustering results of accessing all modes simultaneously.

[0085] By observation Fig. 9It can be seen that although ICSM3 and ICSM4 both achieved good results, ICSM2 of the embodiment of the present invention lags behind MVGC when compared with it. This may be because when considering historical information, the streaming modality method inherits the information retained after step-by-step processing, and its information volume may be slightly insufficient compared with the actual data. Therefore, there is a certain gap in the results.

[0086] Ablation experiments are performed on all datasets and are demonstrated in detail using the Yale and MSRC-v1 datasets as examples, aiming to evaluate the importance of the second and third components of the overall objective function shown in relation (10), namely the discriminative learning part (ICSM-G) and the clustering part (ICSM-B). Fig.10 It can be seen that on different data sets, as new modalities continue to emerge, the method of the embodiment of the present invention consistently achieves the best results, highlighting the key role of each component in the incremental clustering model, and clearly showing that all components need to be integrated to obtain the best results; after removing the discriminant learning items used to highlight the differences between modalities and within modalities in the flow modal process, the performance of ICSM-G has dropped significantly, indicating the importance of distinguishing and classifying external and internal patterns; after removing the clustering items used to limit the number of clusters, the clustering results based on ACC show that the performance of ICSM-B is always lower than that of ICSM. This observation emphasizes the importance of enforcing clustering rank constraints.

[0087] The total objective function involves three parameters: λ, β and γ, and the selection range is {10 -3 ,10 -2 ,…,10 2 ,10 3 Taking ICSM4 as an example (the results of ICSM1-ICSM3 are similar), the parameter sensitivity analysis of the ICSM model is performed on three different datasets, Yale, Caltech101-7, and Handwritten, while keeping β, γ, and λ fixed in all modes. The accuracy (ACC) results are shown in Figure 2. Fig.11As shown, where Figures (a)-(b) represent fixed parameters β, Figures (c)-(d) represent fixed parameters γ, and Figures (e)-(f) represent fixed parameters λ. Figures (a)-(b) and (e)-(f) show that the impact of γ is more significant than λ and β. This observation strongly supports the effectiveness of learning the differences within and between different modes during incremental clustering. In addition, based on the observations of (c)-(d), the impacts of λ and β are relatively similar, which means that the clustering connectivity requirements and modal weight constraints play the same role. The incremental clustering method based on flow modality of an embodiment of the present invention appears to be stable in terms of parameters. When γ, λ and β are adjusted within a certain range, the performance of the method of an embodiment of the present invention changes only slightly. Therefore, this alleviates the challenge of parameter tuning.

[0088] In summary, the incremental clustering method based on streaming modality of an embodiment of the present invention obtains the new modal information of multiple view data samples at the current moment and the historical inheritance comprehensive information of all modalities before the current moment; for any sample, similarity decomposition and discriminant learning are performed on the new modal information and the historical inheritance comprehensive information to obtain the overall objective function; the similarity decomposition result of the new modal information and the historical inheritance comprehensive information that satisfies the overall objective function is solved, and the historical inheritance comprehensive information including the current moment is calculated according to the similarity decomposition result; multiple samples are clustered according to the historical inheritance comprehensive information including the current moment of each sample to obtain clustering results, and the clustering problem of streaming modal scenarios is studied for the first time, which can accurately and effectively perform incremental clustering on streaming modal samples.

[0089] The above specific embodiments of the present invention are described. In some cases, the actions or steps recorded in the embodiments of the present invention can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0090] Based on the same concept, the embodiment of the present invention also provides an incremental clustering device based on flow mode. Applied to a server. Fig.12 As shown, the incremental clustering based on flow mode includes: a mode information acquisition unit, an objective function acquisition unit, an objective solution unit and a clustering unit. Among them,

[0091] A modality information acquisition unit, used to acquire new modality information of multiple view data samples at the current moment and historical inheritance comprehensive information of all modalities before the current moment;

[0092] An objective function acquisition unit, used for performing similarity decomposition and discriminative learning on the new modal information and the historical inheritance comprehensive information for any sample to acquire an overall objective function;

[0093] A target solving unit, used for solving a similarity decomposition result of the new modal information and the historical inheritance comprehensive information that satisfies the overall target function, and calculating the historical inheritance comprehensive information including the current moment according to the similarity decomposition result;

[0094] The clustering unit is used to cluster the multiple samples according to the historical inheritance comprehensive information of each sample including the current moment, and obtain a clustering result.

[0095] For the convenience of description, the above device is described as various modules according to their functions. Of course, when implementing the embodiment of the present invention, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0096] The device of the above embodiment is applied to the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0097] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the above embodiments is implemented.

[0098] An embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the method described in any of the above embodiments.

[0099] Fig.13 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1301, a memory 1302, an input / output interface 1303, a communication interface 1304, and a bus 1305. The processor 1301, the memory 1302, the input / output interface 1303, and the communication interface 1304 are connected to each other in communication within the device through the bus 1305.

[0100] The processor 1301 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided by the method embodiment of the present invention.

[0101] The memory 1302 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1302 may store an operating system and other application programs. When the technical solution provided by the method embodiment of the present invention is implemented by software or firmware, the relevant program code is stored in the memory 1302 and is called and executed by the processor 1301.

[0102] The input / output interface 1303 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0103] The communication interface 1304 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0104] The bus 1305 includes a path that transmits information between the various components of the device (eg, the processor 1301 , the memory 1302 , the input / output interface 1303 , and the communication interface 1304 ).

[0105] It should be noted that, although the above device only shows the processor 1301, the memory 1302, the input / output interface 1303, the communication interface 1304 and the bus 1305, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present invention, and does not necessarily include all the components shown in the figure.

[0106] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0107] This application is intended to cover all such substitutions, modifications and variations that fall within the broad scope of all embodiments. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the scope of protection of this disclosure.

Claims

1. A flow modality-based incremental clustering method, characterized in that: The method comprises: Obtaining new modality information of multiple view data samples at the current moment and historical inheritance comprehensive information of all modalities before the current moment; For any sample, similarity decomposition and discriminative learning are performed on the new modal information and the historical inherited comprehensive information to obtain an overall objective function; Solving the similarity decomposition result of the new modal information and the historical inheritance comprehensive information that satisfies the overall objective function, and calculating the historical inheritance comprehensive information including the current moment according to the similarity decomposition result; Clustering multiple samples according to the historical inheritance comprehensive information of each sample including the current moment, to obtain a clustering result; The performing similarity decomposition and discriminative learning on the new modal information and the historical inheritance comprehensive information to obtain the overall objective function includes: Performing similarity decomposition on the new modal information and the historical inheritance comprehensive information to obtain a first objective function; Performing discriminative learning on the new modal information and the historical inherited comprehensive information from inter-modality and intra-modality to obtain a second objective function; Combining the first objective function and the second objective function to obtain an overall objective function; The performing similarity decomposition on the new modal information and the historical inheritance comprehensive information to obtain a first objective function includes: Decomposing the new modality information into a consistency information matrix of all modalities before the current moment and specific information of the new modality; Decomposing the historical inheritance comprehensive information into the consistency information matrix and specific information of historical modality; Obtaining a first objective function of similarity decomposition according to the consistency information matrix, the specific information of the new modality and the specific information of the historical modality in combination with the real-time update characteristics of the streaming modality data; The discriminative learning of the new modal information and the historical inherited comprehensive information from inter-modality and intra-modality to obtain a second objective function includes: Obtaining a first determination result according to the difference in information carried by different modalities, wherein the first determination result is that there is a significant difference between the specific information of the new modality and the specific information of the historical modality; The similarity matrix is ​​obtained by performing similarity learning on the modalities; Obtaining a second determination result according to different meanings of different attributes of internal information of a single modality, wherein the second determination result is that there are significant differences between the similarity matrix and the specific information of the new modality and the specific information of the historical modality; Acquire a second objective function of discriminative learning based on the first determination result and the second determination result; The combining the first objective function and the second objective function to obtain the total objective function comprises: Solving the trace of the Laplacian matrix corresponding to the consistency information that is minimized according to the flow modal clustering property; Obtaining a total objective function according to the first objective function, the second objective function and the trace of the minimized Laplacian matrix; The overall objective function is: Among them, A T is the consistency information, λ, β, γ are weight parameters, α t is the coefficient at time t, and α is the T and α T-1 The relevant vector, S T is the new modal information at time T, P T is the specific information of the new modality at time T, S T-1 is the historical modal information before time T, P T-1 is the specific information of the historical mode before time T, L A is the Laplace matrix corresponding to the consistency information, and F is the label matrix.

2. The method according to claim 1, characterized in that The calculating of the historical inheritance comprehensive information including the current moment according to the similarity decomposition result includes: The historical inheritance comprehensive information including the current moment is calculated according to the similarity decomposition result using the following relational expression: in, A is the historical inheritance comprehensive information including the current moment. T is the consistency information, P T is the specific information of the new modality at time T, P T-1 It is the specific information of the historical modality before the current time T.

3. An incremental clustering device based on flow mode, characterized in that: The device comprises: A modality information acquisition unit, used to acquire new modality information of multiple view data samples at the current moment and historical inheritance comprehensive information of all modalities before the current moment; An objective function acquisition unit, used for performing similarity decomposition and discriminative learning on the new modal information and the historical inheritance comprehensive information for any sample to acquire an overall objective function; A target solving unit, used for solving a similarity decomposition result of the new modal information and the historical inheritance comprehensive information that satisfies the overall target function, and calculating the historical inheritance comprehensive information including the current moment according to the similarity decomposition result; The clustering unit is used to cluster the multiple samples according to the historical inheritance comprehensive information of each sample including the current moment, and obtain a clustering result.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to claim 1 or 2 is implemented.

5. A computer storage medium, characterized in that: The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute the method according to claim 1 or 2.

Images