Deep adversarial multi-modal data clustering method

Abstract

The invention discloses a deep adversarial multi-modal data clustering method. The method comprises: constructing a modal coding network, a modal fusion network, a modal generator and a modal fusion discriminator; firstly, mapping each modal of data to a depth feature space through a corresponding modal coding network by a model, and learning private depth features of each modal; then, by the modal fusion network, learning private features of each modal, and obtaining fusion features with modal disturbance; and finally, generating, by the modal generator, a sample by using the fusion features, and determining, by the modal fusion discriminator, the authenticity of the sample, so that the modal generator and the modal fusion discriminator fit data distribution through a generative adversarial strategy, and correspondingly designing an adversarial cyclic consistent clustering loss function to guide the training of the model, wherein the adversarial cyclic consistent clustering loss function comprises a cyclic consistent loss function, a cross-modal adversarial loss function and a clustering embedding loss function. According to the deep adversarial multi-modal data clustering method, fusion features of data are learned, and internal rules of the data are mined; and semantic consistency information hidden in the multi-modal data can be effectively extracted, and the multi-modal data clustering performance is improved.

Description

technical field [0001] The invention belongs to the technical field of multimodal data clustering and data mining, and relates to a method for deep confrontation multimodal data clustering. Background technique [0002] With the widespread use of computing devices and multimedia technologies, a large amount of multimodal data has emerged in the real world. Multimodal data is usually composed of multiple data types. The private information of each type of data contains specific information related to the type and consistent semantic information. For example, video is composed of audio and image data, where audio data contains timing information, and image data has Topological information, which together describe the content in the video. Joint analysis of multimodal data to gain knowledge of the fusion of modalities in the data has attracted widespread attention. [0003] Multimodal clustering algorithm, as a basic multimodal data analysis method, uses the similarity betwee...

AI Technical Summary

Problems solved by technology

[0004] However, the current multimodal clustering method only uses the local information reconstructed by the data sample itself for clustering, does not consider the global information implicit in the data distribution, and ignores the semantic consistency information between modalities

In addition, existing methods generally mine data clustering patterns approximately using a weak clustering loss function

The insufficiency of existing multimodal clustering algorithms severely limits the performance of clustering models. Therefore, it is an urgent need to design new methods to effectively explore the inherent laws of multimodal data.

Images