Continuous multi-modal clustering method insensitive to modal arrival sequence

Abstract

The invention relates to a continuous multi-modal clustering method insensitive to a modal arrival sequence, and the method comprises the steps: obtaining multi-modal data, obtaining a high-level feature through the multi-modal data, and carrying out the fusion of the high-level feature and a historical modal feature, and obtaining a fusion feature; inputting the fusion features into a clustering network model, and obtaining a stable clustering distribution result irrelevant to the sequence; the clustering network model is obtained through training of a training set; in the training process, cross-time consistency migration is set, consistency constraint is applied to modal features and clustering distribution at the current moment, and clustering distribution and feature embedding are subjected to comparative learning by utilizing comparative loss. According to the method, accurate and effective incremental clustering can be carried out on the samples of the flow modes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of continuous learning and multi-modal clustering, in particular to a continuous multi-modal clustering method insensitive to the order of modal arrival. BACKGROUND

[0002] In the process of continuous multi-modal clustering, it is necessary to fuse historical modal and newly arrived modal information. However, the existing technology generally lacks reliability modeling, and mostly uses simple splicing or fixed weight linear fusion method, assuming that the modal is homogeneous and the signal-to-noise ratio is consistent, neither evaluating the uncertainty of new modal nor lacking redundancy suppression mechanism. In continuous updating, this will continuously inject noise and irrelevant information into the shared representation, causing the clustering discriminability to decline over time and making it difficult to maintain stable performance. At the same time, the existing technology is highly dependent on the order of modal arrival, especially when high-quality modal arrives first and low-quality modal arrives later. In the distillation process, the modal is treated equally, making low-quality information and high-quality information equally preserved, thereby exacerbating catastrophic forgetting.

[0003] In video monitoring, medical imaging and multi-source sensor analysis scenarios, modal data often arrives in a non-fixed order, and if there is no effective modeling, it will lead to degradation of clustering effect and affect behavior recognition, lesion detection or environmental monitoring tasks.

[0004] The prior art discloses an incremental clustering method and device based on stream modal, which obtains new modal information of a plurality of view data samples at a current time and historical inheritance comprehensive information of all modal before the current time; for each sample, similarity decomposition and discriminative learning are performed on the new modal information and the historical inheritance comprehensive information to obtain a total objective function; the similarity decomposition result of the new modal information and the historical inheritance comprehensive information satisfying the total objective function is solved, and the historical inheritance comprehensive information including the current time is calculated according to the similarity decomposition result; a plurality of samples are clustered according to the historical inheritance comprehensive information including the current time of each sample to obtain a clustering result. The disadvantages of this scheme are: 1. Only incremental clustering of stream modal is processed, and the order bias problem is not considered; 2. Only similarity decomposition and discriminative learning are based, and such structure is lacking; 3. Only simple inheritance update is made; 4. The modal quality selection mechanism is not considered.

[0005] Therefore, there is an urgent need for a continuous multi-modal clustering method that is insensitive to the order of modal and can preferentially retain high-quality information to ensure long-term stable clustering performance. SUMMARY

[0006] To solve the problems in the prior art, the application aims to provide a continuous multi-modal clustering method insensitive to modal arrival order, which explicitly designs a sequence-independent continuous clustering mechanism and can offset the influence of modal arrival order on the result; the learned discriminative subspace is preserved through a residual channel, and new information is accepted with a high-rank global basis, thereby avoiding the bias caused by new modal update; bidirectional alignment and redundancy filtering between the historical fusion representation and the new modal feature are performed to retain complementary information and suppress irrelevant components; knowledge boundary drift is avoided through cross-time consistency transfer, and high-quality modal contributions are preferentially retained in combination with quality-aware historical consolidation.

[0007] To achieve the above object, the application provides the following scheme: A continuous multi-modal clustering method insensitive to modal arrival order, comprising: Obtaining multi-modal data, using the multi-modal data to obtain high-level features, and fusing the high-level features with historical modal features to obtain fused features; Inputting the fused features into a clustering network model to obtain stable and sequence-independent clustering assignment results; the clustering network model is obtained by training using a training set; In the training process, cross-time consistency transfer is set to impose consistency constraints on the modal features and clustering assignments at the current time, and the clustering assignments are compared with feature embeddings for contrast learning using a contrast loss.

[0008] Optionally, obtaining the high-level features comprises: Inputting the multi-modal data into a feature encoder network model to obtain latent features, and mapping the latent features into the high-level features through a multi-layer perceptron.

[0009] Optionally, obtaining the fused features comprises: Using a residual fusion network model to synthesize the historical modal features into global base features: ; Wherein, is the global base feature, is the modal feature information, is the residual fusion network. Further fusing the high-level features and the global base features to obtain the fused features: ; Wherein, is the fused feature, is the feature representation of the modal arrived at the tth time.

[0010] Optionally, in the process of fusing the high-level features and the global basic features, a cross-time knowledge coordination mechanism is adopted to perform bidirectional information screening between the global basic features and the high-level features, and to obtain complementary effective information. The loss function of the cross-time knowledge coordination mechanism comprises: is the fused feature, i.e., the fused feature, is the historical fused information, i.e., the global basic feature, is the newly arrived modal feature information, i.e., the high-level feature, represents a sample centering operation, which is used to calculate a zero-mean cross-covariance matrix to avoid the interference of the mean term, and MCE represents a matrix cross-entropy, is a d is a d-dimensional unit matrix, is a feature dimension, is a batch size, is a transpose, denotes historical fused information , is the fused feature .

[0011] Optionally, the loss function of the cross-time consistency migration comprises: is the loss function of the cross-time consistency migration, represents the modal feature and the cluster assignment at the current time, represents the modal feature and the cluster assignment at the historical time, is obtained by the fused feature through a clustering network, is stabilized by using the global cluster assignment Q as a supervision signal.

[0012] Optionally, the contrastive loss comprises: is the contrastive loss represents a cosine similarity, is a temperature hyperparameter, is a negative set, , is a sample matrix, is a subset of the negative sample set a, is an exponential function.

[0013] Optionally, the method further comprises: ​​​​​​determine a loss function based on quality perception history consolidation through the contrast loss: ; wherein, is the loss function based on quality perception history consolidation, is an entropy of cluster assignment, is an adaptive weight, is the contrast loss.

[0014] Optionally, obtaining the adaptive weight comprises: determining a distribution similarity between modal features in the feature embedding and the fusion feature: ; wherein, is the distribution similarity, is a Gaussian kernel, is a modal feature in a feature embedding of an i-th batch size, is a modal feature in a feature embedding of a j-th batch size, is a fusion modal feature of a j-th batch size, is a fusion modal feature of a j'-th batch size; mapping the distribution similarity to a quality score and normalizing to obtain the adaptive weight: ; ; ; wherein, is a closeness degree at a k-th time point, is a time variable.

[0015] Optionally, the method further comprises: optimizing the overall loss function through an ADAM optimizer and a back propagation strategy.

[0016] The present application has the following advantages: ​​​​The application can ensure reliable admission of historical information and new information by suppressing update bias and retaining learned discriminative subspace when new modalities arrive through the residual fusion network; can perform bidirectional alignment and information filtering between historical fusion representation and new modality features through the cross-time knowledge synergy mechanism, on the one hand to retain complementary effective information, and on the other hand to suppress redundant and task-irrelevant components from entering the shared representation, thereby reducing temporal cumulative noise and improving clustering discriminativeness; can maintain consistency between current representation and historical representation through cross-time consistency migration, achieving smooth inheritance and boundary stability of knowledge; can perform preferential retention and reinforcement of high-quality modality features and cluster assignments through quality-aware historical consolidation, and suppress low-quality modality disturbance, thereby reducing the risk of forgetting caused by modality arrival order. Through the synergistic effect of the above technical features, the application realizes reliable fusion of historical and current information and offset of modality order bias, achieving long-term stable and sequence-independent clustering performance. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 A flow chart of a modality arrival order insensitive continuous multi-modality clustering method according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0020] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0021] As Figure 1As shown, the embodiment discloses a continuous multi-modal clustering method insensitive to modal arrival order, comprising: acquiring multi-modal data, obtaining high-level features from the multi-modal data, and fusing the high-level features with historical modal features to obtain fused features; inputting the fused features into a clustering network model to obtain stable and sequence-independent clustering assignment results; the clustering network model is obtained by training with a training set; in the training process, a cross-time consistency migration is set to impose consistency constraints on the modal features and clustering assignment at the current time, and contrastive learning is performed on the clustering assignment and feature embedding by using a contrastive loss.

[0022] The embodiment discloses a continuous multi-modal clustering method insensitive to modal arrival order, comprising: first, a reliable continuous information propagation framework is realized by suppressing the update bias brought by new modal and maximizing the retention of task-related information, second, the modal sequence insensitivity is realized by reliable inheritance of historical knowledge and high-quality modal information retention (step four), and then the clustering assignment insensitive to modal sequence is finally obtained by optimizing the overall loss function.

[0023] At the Tth time, multi-modal input The latent features are obtained through encoder-decoder pre-training , which are mapped into high-level features by a multi-layer perceptron . The modal before T-1 time is historical information, and the fusion is obtained . The high-level features are fused with to obtain , and finally is sent to a clustering network to obtain the final assignment , wherein the clustering network is composed of three fully connected neural networks, and the output dimension of the last layer is the number of clusters. Take the medical image field as an example: Suppressing the update bias brought by new modal: in the medical image field, the acquisition results of different images (such as CT, MRI, PET, etc.) have time sequence, and there are significant differences in resolution and sensitivity to tissue structure. If the newly arrived image modal is directly fused, it is easy to cause the drift of the learned discriminative subspace. Therefore, a residual fusion network is used to suppress the update bias brought by the new modal, the learned discriminative subspace is retained through the residual channel, and the new modal information is accepted in the form of high-rank global basis. The residual fusion network is: (1) Wherein, M represents splicing multi-modal information, , , B represents the size of the batch, d represents the feature dimension, represents the real set.

[0024] The residual fusion network operates in two stages. First, multiple historical modalities are fused into a stable global basis: (2) Second, the fusion is combined with the newly arrived modalities: (3) Maximizing task-related information: In medical images, the information contained in different imaging modalities is complementary, such as CT focusing more on bone tissue structure, while MRI focusing more on soft tissue signals. This step proposes a cross-time knowledge coordination mechanism to perform bidirectional information filtering between the historical fusion representation and the new modality high-level features (where new represents the current newly arrived modality, and t represents the Tth time). On the one hand, it maximizes the retention of complementary effective information from both, and on the other hand, it suppresses task-irrelevant components from entering the shared representation with a redundancy reduction constraint, thereby reducing temporal cumulative noise and improving balanced utilization and discriminability of features. This mechanism includes two channels and , and its loss function is: (4) where is the fused feature, is the historical fusion information, is the newly arrived modality feature information, denotes sample centering operation, which is used to calculate the zero-mean cross-covariance matrix to avoid the interference of the mean term. MCE represents the matrix cross-entropy.

[0025] Reliable inheritance of historical knowledge: As new medical images are continuously added, if there is a lack of historical knowledge constraints, the model is prone to "boundary drift", leading to degradation of the diagnostic ability of early modalities. Therefore, cross-time consistency transfer is designed to impose consistency constraints on the current time representation, making it close to the previous time in terms of both features and clustering distribution, thereby smoothly inheriting historical knowledge and avoiding boundary drift. The loss of this part is: (5) where the first term enforces the temporal smoothness of the features and assignments, and the KL term uses the global clustering assignment Q as a supervision signal to stabilize the clustering to prevent drift. denotes the modality feature and clustering assignment at the current time, denotes the modality feature and clustering assignment at the historical time. is obtained by a clustering network from the global fusion feature .

[0026] Effective preservation of high-quality modality history information: Different imaging modalities have differences in signal-to-noise ratio, scanning parameters or lesion coverage. If low-quality modalities are preserved indiscriminately, it may introduce error feature migration. Quality-aware history consolidation: suppress low-quality modality disturbance, thereby reducing the path dependence and forgetting risk caused by the arrival order. For any pair of modalities , the present application performs contrast learning on feature embedding and cluster assignment . Among them, , , is the modality feature representation at the t-th time, is the cluster representation at the t-th time, (·) and (·) are feature encoder network and clustering network respectively. The contrast loss is: (6) where, denotes the cosine similarity, is the temperature, is the negative set.

[0027] Therefore, the loss based on quality-aware history consolidation is: (7) where, is the entropy of cluster assignment, is the probability that sample is assigned to cluster in modality . Maximizing the entropy (through term) can suppress degenerative solutions while keeping the new modality consistent with the historical semantic space.

[0028] The adaptive weight originates from the distribution similarity between modality feature and fusion representation . Using Gaussian kernel , the empirical square MMD is: (8) The present application maps it to the quality score and normalizes it: (9) The modality that is more consistent with the fusion representation obtains a larger weight, thereby making a greater contribution to integration.

[0029] Overall loss function optimization: optimize the overall loss function through the ADAM optimizer and the back propagation strategy, so as to realize stable and sequence-independent cluster assignment.

[0030] Obtain a modal arrival order-insensitive clustering division: through reliable inheritance of historical knowledge and effective preservation of high-quality modal historical information, an optimized Finally, it is passed to the clustering network for stable and sequence-independent clustering assignment.

[0031] The application provides a method for bidirectional alignment and information screening of historical fusion representation and new modal features through a cross-time knowledge cooperation mechanism, which can inhibit the introduction of redundant components, reduce the accumulation of time noise, and improve the balance of feature utilization and clustering discrimination ability. The application provides a method for joint modeling of cross-time consistency migration and quality perception based on modal importance scoring through an adaptive anti-forgetting strategy, which can preferentially retain high-value historical information and inhibit the interference of low-quality modal, thereby effectively reducing path dependence and catastrophic forgetting, and realizing the clustering performance insensitive to the modal arrival order.

[0032] The above-described embodiments are only descriptions of the preferred modes of the application and do not limit the scope of the application. Without departing from the design spirit of the application, various modifications and improvements to the technical solutions of the application made by those skilled in the art shall fall within the protection scope determined by the claims of the application.

Claims

1. A persistent multi-modal clustering method that is insensitive to modal arrival order, characterized in that, The method comprises: acquiring multi-modal data, obtaining high-level features using the multi-modal data, and fusing the high-level features with historical modal features to obtain fused features; inputting the fused features into a clustering network model to obtain stable and sequence-independent clustering assignment results; the clustering network model is obtained by training using a training set; during the training process, a cross-time consistency migration is set to impose consistency constraints on the modal features and clustering assignments of the current time, and a contrastive loss is used to perform contrastive learning on the clustering assignments and feature embeddings.

2. The persistent multi-modal clustering method not sensitive to modal arrival order according to claim 1, characterized in that, obtaining the high-level features comprises: inputting the multi-modal data into a feature encoder network model to obtain latent features, and mapping the latent features into the high-level features through a multi-layer perception machine.

3. The method, as claimed in claim 1, wherein the method is not sensitive to the order of arrival of the modalities. acquiring the fused features comprises: using a residual fusion network model to synthesize the historical modal features into global basic features: ； wherein, is a global base feature, is a modal feature information, is a residual fusion network; further fusing the high-level features and the global basic features to obtain the fused features: ； wherein, is the fused feature, is the feature representation of the modality at the t-th time instant.

4. The persistent multi-modal clustering method not sensitive to modal arrival order according to claim 3, characterized in that, during the fusion of the high-level features and the global basic features, a cross-time knowledge collaboration mechanism is used to perform bidirectional information screening between the global basic features and the high-level features to obtain complementary effective information; the loss function of the cross-time knowledge collaboration mechanism comprises: ； wherein, is the fused feature, i.e., the fusion feature, is the history fusion information, i.e., the global base feature, is the newly arrived modality feature information, i.e., the high-level feature, denotes a sample centering operation for computing a zero-mean cross-covariance matrix to avoid mean term interference, and MCE denotes a matrix cross-entropy, is a d is a d-dimensional identity matrix, is the feature dimension, is the batch size, is a transpose, denotes the history fusion information , is the fused feature .

5. The method, as claimed in claim 1, wherein the method is not sensitive to the order of arrival of the modalities. the loss function of the cross-time consistency migration comprises: ； wherein, is the loss function for cross-time consistency migration, denotes the modal feature and cluster assignment at the current time, denotes the modal feature and cluster assignment at the historical time, is obtained by the clustering network, is to stabilize the clustering by using the global cluster assignment Q as a supervision signal.

6. The method, as claimed in claim 1, wherein the method is not sensitive to the order of arrival of the modalities. the contrastive loss comprises: ； wherein, is a contrastive loss denotes a cosine similarity, is a temperature hyperparameter, is a negative set, , is a sample matrix, is a subset of a negative sample set, is an exponential function.

7. The persistent multi-modal clustering method not sensitive to modal arrival order according to claim 6, characterized in that, The method further comprises: determining a loss function based on quality-aware historical consolidation through the contrastive loss: ； wherein, is a loss function based on quality perception history reinforcement, is an entropy of cluster assignment, is an adaptive weight, is a contrastive loss.

8. The persistent multi-modal clustering method not sensitive to modal arrival order according to claim 7, characterized in that, acquiring the adaptive weight comprises: determining the distribution similarity between the modal features in the feature embedding and the fused features: ； wherein, is a distribution similarity, is a Gaussian kernel, is a modality feature in the feature embedding of the th batch size, is a modality feature in the feature embedding of the th batch size, is a fused modality feature of the jth batch size, is a fused modality feature of the j'th batch size; mapping the distribution similarity to a quality score and normalizing it to obtain the adaptive weight: ； ； wherein, is the time variable. at a time t.

9. The method, as claimed in claim 1, wherein the method is not sensitive to the order of arrival of the modalities. The method further comprises optimizing the overall loss function through an ADAM optimizer and a backpropagation strategy.

Images