Nonholonomic multimodal deep clustering method, device and computer equipment based on latent representation

Abstract

The application relates to a non-complete multi-modal deep clustering method, device and computer equipment based on a hidden representation. The method comprises the following steps: respectively mapping hidden representations of samples to feature spaces of corresponding modes to obtain reconstructed data representations of the samples in the modes, obtaining missing modes of the samples according to mode missing indication vectors, completing missing mode data by using the reconstructed data representations of the missing modes to obtain complete multi-modal data, performing kernel alignment on linear kernel matrices of the multi-modal data and the hidden representations of the samples in the complete multi-modal data, obtaining a data alignment loss according to kernel alignment losses of the modes, constructing a data reconstruction loss according to distances between the reconstructed data representations and corresponding mode data, optimizing the data reconstruction loss and the data alignment loss, and performing clustering on the optimized hidden representations of the samples to obtain data labels of each classification. The method can improve the efficiency and accuracy of non-complete multi-modal deep clustering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a non-complete multi-modal deep clustering method based on hidden representation, device and computer equipment. BACKGROUND

[0002] With the rapid development of electronic equipment and information technology, multi-modal data has become increasingly common in real-world applications. Multi-modal data covers information in multiple dimensions of the same batch of things, such as multi-modal information such as user's living habits, shopping records in Internet advertising recommendation scenarios, and multi-modal data such as blood tests and image detection of patients in Alzheimer's disease diagnosis. Based on this, multi-modal clustering technology has emerged, which can fuse and analyze multiple modal data without label information and classify them, becoming a research hotspot.

[0003] At present, most of the existing multi-modal clustering methods are modified from single-modal clustering and mainly divided into five categories such as multi-kernel clustering and multi-modal subspace clustering. Among them, multi-modal deep clustering performs outstandingly in clustering performance by virtue of the powerful learning ability of neural networks, which usually maps each modal data to independent hidden representation using an encoder, and then performs fusion clustering, and the two processes assist each other. To further improve the performance, some methods also use more powerful neural networks such as convolutional neural networks and Transformers. When facing the problem of missing modalities caused by sensor damage or environmental restrictions in reality, non-complete multi-modal deep clustering methods are proposed. This kind of method also maps visible modal data to hidden representation, and then performs fusion clustering, while introducing loss function and constraint term to optimize the clustering results.

[0004] However, the existing non-complete multi-modal deep clustering method has obvious shortcomings, which hinders the further improvement of clustering performance. First, it ignores the uniqueness of data hidden representation in each modality, independently processes the hidden representation corresponding to different modal data, and does not realize that the potential representation should have uniqueness; second, it ignores the similarity between samples in the missing modalities, only fuses the hidden representation corresponding to the existing modal data, and does not consider the missing data. SUMMARY

[0005] Therefore, it is necessary to provide a non-complete multi-modal deep clustering method based on hidden representation, device and computer equipment to solve the above technical problems.

[0006] A non-complete multi-modal deep clustering method based on hidden representation, the method comprising:

[0007] obtaining multi-modal data, modality missing indication vector and classification number; the multi-modal data comprises N V ​modal data; the modal missing indication vector is used to indicate the first v The modal is missing or visible;

[0008] initialization N The latent representation of samples is used V A neural network maps the latent representation of each sample to the feature space of the corresponding modality, obtains the reconstructed data representation of each modality of the sample, obtains the missing modality of each sample according to the modality missing indicator vector, uses the reconstructed data representation of the missing modality to complete the missing modality data, and obtains the complete multimodal data based on the visible modality data of each sample and the completed missing modality data;

[0009] Perform kernel alignment on the linear kernel matrix of each modal data and each sample implicit representation in the complete multimodal data, and perform weighted fusion of the kernel alignment loss of each modality to obtain the data alignment loss;

[0010] Constructing a data reconstruction loss according to the distance between each reconstructed data representation and the corresponding modality data, and obtaining a total loss function according to the data reconstruction loss and the data alignment loss;

[0011] The total loss function is optimized, and the optimized latent representations of the samples are used for clustering to obtain data labels for each classification category.

[0012] In one embodiment, the method further includes constructing a data reconstruction loss based on the distance between each reconstructed data representation and the corresponding modal data:

[0013]

[0014] in, is the data reconstruction loss, is the number of samples, is the number of modes, For the The number of visible modes for a sample, is the modal missing indicator vector, For the The first sample Reconstructed data representation of modal data, For the The first sample Modal data, is the Euclidean distance.

[0015] In one embodiment, the method further includes: calculating the linear kernel matrix of the latent representation of each sample in the complete multimodal data to obtain the latent representation kernel matrix; calculating the linear kernel matrix of each modality in the complete multimodal data to obtain the modal kernel matrix corresponding to each modality; and aligning the kernel matrices by maximizing the similarity between the latent representation kernel matrix and the kernel matrices of each modality.

[0016] In one of the embodiments, the data alignment loss is obtained by weighting and fusing the core alignment losses of the modalities.

[0017] ;

[0018] wherein, is the data alignment loss, is the weight of the th modality, is the core alignment loss of the th modality.

[0019] In one of the embodiments, the data reconstruction loss and the data alignment loss are optimized to obtain the optimized neural network parameters, the latent representations of the samples, and the weight parameters of the modalities.

[0020] In one of the embodiments, the reconstructed data representation is:

[0021] ;

[0022] wherein, is the reconstructed data representation of the th sample and the th modality, is the latent representation of the th sample, represents the parameter of the th neural network.

[0023] In one of the embodiments, the total loss function is:

[0024]

[0025] wherein, is the total loss function, is the data reconstruction loss, is the data alignment loss, is a hyperparameter.

[0026] An apparatus for nonholonomic multi-modal deep clustering based on latent representation, comprising:

[0027] a data acquisition module, configured to acquire multi-modal data, a modality missing indication vector, and a number of classes; the multi-modal data comprises N modal data of V samples; the modality missing indication vector is used to indicate whether the v th modality of the sample is missing or visible;

[0028] a data reconstruction module, configured to initialize Nhidden representations of the samples, by using V the neural networks respectively map the hidden representations of the samples to feature spaces of corresponding modalities to obtain reconstructed data representations of the samples in the modalities, obtain missing modalities of the samples according to the modality missing indication vectors, complete missing modality data by using the reconstructed data representations of the missing modalities, and obtain complete multi-modality data according to the visible modality data of the samples and the completed missing modality data;

[0029] a data alignment module configured to perform kernel alignment on linear kernel matrices of the modality data in the complete multi-modality data and the hidden representations of the samples, and obtain a data alignment loss by weighted fusion of kernel alignment losses of the modalities;

[0030] a loss construction module configured to construct a data reconstruction loss according to distances between the reconstructed data representations and corresponding modality data, and obtain a total loss function according to the data reconstruction loss and the data alignment loss;

[0031] a result output module configured to optimize the total loss function, and perform clustering by using optimized hidden representations of the samples to obtain data labels of the classes.

[0032] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0033] obtain multi-modality data, a modality missing indication vector, and a number of classes; the multi-modality data includes N hidden representations of the samples, by using V the neural networks respectively map the hidden representations of the samples to feature spaces of corresponding modalities to obtain reconstructed data representations of the samples in the modalities, obtain missing modalities of the samples according to the modality missing indication vector, complete missing modality data by using the reconstructed data representations of the missing modalities, and obtain complete multi-modality data according to the visible modality data of the samples and the completed missing modality data; v

[0034] initialize N hidden representations of the samples, by using V the neural networks respectively map the hidden representations of the samples to feature spaces of corresponding modalities to obtain reconstructed data representations of the samples in the modalities, obtain missing modalities of the samples according to the modality missing indication vector, complete missing modality data by using the reconstructed data representations of the missing modalities, and obtain complete multi-modality data according to the visible modality data of the samples and the completed missing modality data;

[0035] perform kernel alignment on linear kernel matrices of the modality data in the complete multi-modality data and the hidden representations of the samples, and obtain a data alignment loss by weighted fusion of kernel alignment losses of the modalities;

[0036] construct a data reconstruction loss according to distances between the reconstructed data representations and corresponding modality data, and obtain a total loss function according to the data reconstruction loss and the data alignment loss;

[0037] ​Optimizing the total loss function, clustering by using the optimized hidden representation of each sample to obtain the data label of each classification.

[0038] A computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the following steps:

[0039] Obtaining multi-modal data, a modal missing indication vector, and a number of classification categories; the multi-modal data includes N modal data of each sample; the modal missing indication vector is used to indicate whether the i-th modal of each sample is missing or visible; V v

[0040] Initializing the hidden representation of each sample, mapping the hidden representation of each sample to the feature space of the corresponding modal by using i neural networks respectively to obtain the reconstructed data representation of each modal of each sample, obtaining the missing modal of each sample according to the modal missing indication vector, completing the missing modal data by using the reconstructed data representation of the missing modal, and obtaining complete multi-modal data according to the visible modal data and the completed missing modal data of each sample; N V Performing kernel alignment on the linear kernel matrix of each modal data and the hidden representation of each sample in the complete multi-modal data, and obtaining a data alignment loss by weighted fusion of the kernel alignment loss of each modal;

[0041] Constructing a data reconstruction loss according to the distance between each reconstructed data representation and the corresponding modal data, and obtaining a total loss function according to the data reconstruction loss and the data alignment loss;

[0042] Optimizing the total loss function, clustering by using the optimized hidden representation of each sample to obtain the data label of each classification.

[0043] Optimizing the total loss function, clustering by using the optimized hidden representation of each sample to obtain the data label of each classification.

[0044] ​​​The non-complete multi-modal deep clustering method, device and computer equipment based on the hidden representation can provide a reliable basis for the completion of the missing modal data by mapping the multi-modal data to a unified hidden space and accurately reconstructing the visible modal data, and can obtain complete multi-modal data by intelligently completing the missing modal data, thereby effectively solving the problem of insufficient information caused by data missing in non-complete multi-modal data clustering. By aligning the linear kernel matrix of each modal data and the hidden representation and constructing the alignment loss in a weighted fusion manner, the consistency of the similarity of different modal data in the hidden space can be ensured, and the influence of the semantic conflict between the modes on the clustering result is avoided. By integrating the data reconstruction loss and the data alignment loss into a total loss function and optimizing it, the model can learn the reconstruction ability and cross-modal alignment ability of the data at the same time, thereby improving the accuracy and robustness of the clustering. By clustering the optimized hidden representation, the classification result that conforms to the internal structure of the data can be finally obtained. The embodiments of the present application can improve the efficiency and accuracy of non-complete multi-modal deep clustering. BRIEF DESCRIPTION OF DRAWINGS DETAILED DESCRIPTION OF THE INVENTION BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A flowchart of a non-complete multi-modal deep clustering method based on a hidden representation in an embodiment is shown.

[0046] Figure 2 A structural block diagram of a non-complete multi-modal deep clustering device based on a hidden representation in an embodiment is shown.

[0047] Figure 3 An internal structure diagram of a computer equipment in an embodiment is shown. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0049] In one embodiment, as shown in Figure 1 , a non-complete multi-modal deep clustering method based on a hidden representation is provided, comprising the following steps:

[0050] Step 102, obtaining multi-modal data, a modal missing indication vector and a number of classification categories.

[0051] The multi-modal data includes N modal data of V samples. Multi-modal data refers to a collection of different types of features of the same object, such as a student's test scores (numerical), learning logs (text) and online behavior trajectories (sequence). The multi-modal data can be represented as .

[0052] The modality absence indication vector is used to indicate the first v modalities of the samples are absent or visible. The modality absence indication vector is wherein is a discrete vector with values of 0 or 1. For example, represents the first modalities of the first sample are absent, represents the first modalities of the first sample are visible.

[0053] Step 104, initializing N the hidden representations of the samples, mapping the hidden representations of the samples to the feature spaces of the corresponding modalities respectively by using V the neural networks, obtaining the reconstructed data representations of the samples in each modality, obtaining the absent modalities of the samples according to the modality absence indication vector, completing the absent modality data by using the reconstructed data representations of the absent modalities, and obtaining the complete multi-modal data according to the visible modality data and the completed absent modality data of the samples.

[0054] The hidden representation refers to the feature vector after the original high-dimensional data is mapped to a low-dimensional space, and contains the essential features of the original data. For example, various behavior data of a user are mapped to a 128-dimensional vector. Unlike existing methods that map the data of each modality to an independent hidden representation, the present application considers the uniqueness of the data hidden representation under each modality, and therefore the unique hidden representation is denoted as . Wherein , is the dimension of the hidden representation feature.

[0055] The neural network specifically refers to a decoder network used for reconstructing data, and each modality corresponds to an independent network. For example, an LSTM network is used for the text modality, and a CNN network is used for the picture modality. The present application reconstructs the multi-modal data by using an independent neural network in each modality to map the data feature space of the hidden representation. It should be noted that is not a constant determined in advance, but a variable that can be optimized during model training.

[0056] It can be understood that by hidden representation learning, the multi-modal data is unified to the same feature space, and the correlation between modalities is used to complete the absent data. This solves the information loss problem of incomplete multi-modal data when clustering, so that all samples can participate in subsequent analysis with complete feature vectors.

[0057] Step 106, kernel alignment is performed on the linear kernel matrix of each modality data and each sample hidden representation in the complete multi-modal data, and the kernel alignment loss of each modality is weighted and fused to obtain a data alignment loss.

[0058] The linear kernel matrix refers to a similarity matrix calculated by inner product, representing the similarity between samples in a specific modality. Kernel alignment refers to aligning the kernel matrix of different modalities with the kernel matrix of the hidden representation, ensuring that the similarity relationship in the hidden space is consistent with the original modality. For example, similar users in the text modality should also be close in the hidden space.

[0059] It can be understood that by minimizing the alignment loss, the system can learn consistent feature representation across modalities, providing a more reliable basis for clustering. It ensures that the similarity relationship of data in different modalities remains consistent in the hidden space, solving the semantic conflict problem of multi-modal data when fused.

[0060] Step 108, a data reconstruction loss is constructed according to the distance between each reconstructed data representation and the corresponding modality data, and a total loss function is obtained according to the data reconstruction loss and the data alignment loss.

[0061] By balancing the reconstruction loss and the alignment loss, the model can accurately restore the original data and maintain consistency across modalities in the hidden space. This provides a clear direction for subsequent parameter optimization.

[0062] Step 110, the total loss function is optimized, and the hidden representation of each sample obtained by optimization is used for clustering to obtain the data label of each cluster.

[0063] The final hidden representation is obtained by optimization, which not only preserves the features of the original data, but also maintains consistent similarity relationships across modalities. Based on these hidden representations, clustering can obtain more accurate and meaningful classification results. The clustering algorithm can use the K-means algorithm.

[0064] The non-complete multi-modal deep clustering method based on hidden representation proposed by the present application consists of two parts, namely data hidden representation learning and data clustering. Among them, data hidden representation learning can be divided into three modules, namely data reconstruction, data completion and data alignment. At the same time, data clustering is to use k-means clustering on the data hidden representation obtained by optimization to calculate the class division of the data. It can be understood that through data reconstruction and data completion, incomplete data is converted into complete feature vectors, and through kernel alignment and loss function design, it is ensured that the semantics of data in different modalities remain consistent in the hidden space, finally realizing more accurate and robust non-complete multi-modal data clustering effect than traditional methods.

[0065] In the above-mentioned incomplete multimodal deep clustering method based on implicit representation, by mapping multimodal data to a unified latent space and accurately reconstructing visible modal data, a reliable basis can be provided for the completion of missing modal data. By intelligently completing the missing modal data, complete multimodal data can be obtained, effectively solving the problem of insufficient information caused by missing data in incomplete multimodal data clustering. By aligning the linear kernel matrix of each modal data and the latent representation, and constructing the alignment loss in a weighted fusion manner, the similarity consistency of different modal data in the latent space can be ensured, avoiding the influence of semantic conflicts between modalities on the clustering results. By integrating the data reconstruction loss and the data alignment loss into a total loss function and optimizing them, the model can simultaneously learn the data reconstruction ability and cross-modal alignment ability, thereby improving the accuracy and robustness of clustering. Clustering through the optimized latent representation can ultimately obtain a classification result that is more in line with the intrinsic structure of the data. The embodiment of the present invention can improve the efficiency and accuracy of incomplete multimodal deep clustering.

[0066] In one embodiment, constructing the data reconstruction loss according to the distance between each reconstructed data representation and the corresponding modality data includes: constructing the data reconstruction loss according to the distance between each reconstructed data representation and the corresponding modality data as:

[0067] ;

[0068] in, is the data reconstruction loss, is the number of samples, is the number of modes, For the The number of visible modes for a sample, is the modal missing indicator vector, For the The first sample Reconstructed data representation of modal data, For the The first sample Modal data, is the Euclidean distance.

[0069] After reconstructing the data of both visible and missing modalities, the reconstructed data can be used to complete the missing model, that is,

[0070] ;

[0071] Through data completion, complete data can be obtained Based on this, the similarity of data in the missing modality is incorporated into the model training to guide the generation of latent representations.

[0072] In one embodiment, the kernel alignment of the linear kernel matrix of each modality data and the hidden representation of each sample in the complete multi-modal data comprises: calculating the linear kernel matrix of the hidden representation of each sample in the complete multi-modal data to obtain a hidden representation kernel matrix; calculating the linear kernel matrix of each modality in the complete multi-modal data to obtain a modality kernel matrix corresponding to each modality; and performing kernel matrix alignment by maximizing the similarity of the hidden representation kernel matrix and the modality kernel matrix. The corresponding formula can be expressed as:

[0073] ;

[0074] The above formula needs to explicitly calculate kernel matrices, which has high computational complexity and low efficiency. Therefore, it can be equivalently converted to

[0075] ;

[0076] On the basis of the above, the loss function on the th modality can be written as:

[0077] .

[0078] Considering all modalities of data, the loss of data alignment is obtained.

[0079] In one embodiment, the weighted fusion of the kernel alignment loss of each modality to obtain the data alignment loss comprises: the weighted fusion of the kernel alignment loss of each modality to obtain the data alignment loss is:

[0080] ;

[0081] Wherein, is the data alignment loss, is the weight of the th modality, is the kernel alignment loss of the th modality.

[0082] In one embodiment, the reconstructed data representation is:

[0083] ;

[0084] Wherein, is the reconstructed data representation of the th modality of the th sample, is the hidden representation of the th sample, represents the parameters of the th neural network.

[0085] In one embodiment, the total loss function is:

[0086] ;

[0087] wherein, is the total loss function, is the data reconstruction loss, is the data alignment loss, is a hyper-parameter. In the present embodiment, and are hyper-parameters balancing the losses among the modules, which are set in advance.

[0088] In one embodiment, optimizing the total loss function comprises: optimizing the data reconstruction loss and the data alignment loss to obtain the optimized neural network parameters, the latent representations of each sample, and the weight parameters of each modality. In the present embodiment, during the model training process, three variables need to be optimized, i.e., the neural network parameters , the latent representations and the weight parameters . The first two variables are optimized using the common stochastic gradient descent method. The last variable has a closed-form solution, and is:

[0089] .

[0090] In one specific embodiment, the optimization process can be summarized as the pseudo-code in Algorithm 1:

[0091]

[0092] The present method is experimentally verified on the public datasets HandWritten and Flower17. Among them, the comparative algorithms are the DITA-IMVC, DSIMVC, DCP and DVIMC algorithms in the literature, and three missing rates of 0.1, 0.3 and 0.5 are set. The present method is denoted as IMDC, and the clustering accuracy performance obtained by the experiment is shown in Table 1. It can be seen that the IMDC algorithm achieves the best clustering accuracy.

[0093] Table 1 Comparison table of clustering accuracy

[0094]

[0095] It should be understood that, although each step in the flowchart of Figure 1 is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless explicitly stated in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0096] In one embodiment, Figure 2 As shown, a non-complete multimodal deep clustering device based on latent representation is provided, comprising:

[0097] The data acquisition module 202 is used to obtain multimodal data, modality missing indicator vectors and the number of classifications; the multimodal data includes N of samples V modal data; the modal missing indicator vector is used to indicate the first v The modal is missing or visible;

[0098] Data reconstruction module 204, used to initialize N The latent representation of samples is used V A neural network maps the latent representation of each sample to the feature space of the corresponding modality, obtains the reconstructed data representation of each modality of the sample, obtains the missing modality of each sample according to the modality missing indicator vector, uses the reconstructed data representation of the missing modality to complete the missing modality data, and obtains the complete multimodal data based on the visible modality data of each sample and the completed missing modality data;

[0099] The data alignment module 206 is used to perform kernel alignment on the linear kernel matrix of each modal data and each sample implicit representation in the complete multimodal data, and perform weighted fusion of the kernel alignment losses of each modality to obtain the data alignment loss;

[0100] A loss construction module 208 is configured to construct a data reconstruction loss based on the distance between each reconstructed data representation and the corresponding modality data, and obtain a total loss function based on the data reconstruction loss and the data alignment loss;

[0101] The result output module 210 is used to optimize the total loss function, perform clustering using the optimized latent representation of each sample, and obtain data labels for each classification category.

[0102] In one embodiment, the data reconstruction loss is constructed based on the distance between each reconstructed data representation and the corresponding modality data:

[0103]

[0104] in, is the data reconstruction loss, is the number of samples, is the number of modes, For the The number of visible modes for a sample, is the modal missing indicator vector, For the The first sample Reconstructed data representation of modal data, For the The first sample Modal data, is the Euclidean distance.

[0105] In one embodiment, it is also used to calculate the linear kernel matrix of the latent representation of each sample in the complete multimodal data to obtain the latent representation kernel matrix; calculate the linear kernel matrix of each modality in the complete multimodal data to obtain the modal kernel matrix corresponding to each modality; and align the kernel matrices by maximizing the similarity between the latent representation kernel matrix and the kernel matrices of each modality.

[0106] In one embodiment, the weighted fusion of the kernel alignment losses of each modality is used to obtain the data alignment loss:

[0107] ;

[0108] in, is the data alignment loss, For the The weight of each mode, For the Kernel alignment loss for all modalities.

[0109] In one embodiment, it is also used to optimize data reconstruction loss and data alignment loss to obtain optimized neural network parameters, implicit representations of each sample, and weight parameters of each modality.

[0110] In one embodiment, the reconstructed data is represented as:

[0111] ;

[0112] in, For the Sample No. The reconstructed data representation of the modality, For the The latent representation of a sample, Indicates the The parameters of the neural network.

[0113] In one embodiment, the total loss function is:

[0114] ;

[0115] wherein, is a total loss function, is a data reconstruction loss, is a data alignment loss, is a hyper-parameter.

[0116] The specific limitations of the apparatus for nonholonomic multimodal deep clustering based on latent representation can refer to the limitations of the method for nonholonomic multimodal deep clustering based on latent representation described above, which will not be repeated here. Each module in the apparatus for nonholonomic multimodal deep clustering based on latent representation described above can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each of the above modules.

[0117] In one embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in Figure 3 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a method for nonholonomic multimodal deep clustering based on latent representation. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad provided on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0118] Those skilled in the art can understand that, Figure 3 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0119] In one embodiment, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method in the above-mentioned embodiments when executing the computer program.

[0120] In one embodiment, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the steps of the method of the above embodiments.

[0121] A person of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above embodiments. Any reference to memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0122] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0123] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A nonholonomic multimodal deep clustering method based on latent representation, characterized in that, The method comprises: Obtaining multi-modal data, a modal missing indication vector, and a classification number; the multi-modal data includes N V modal data of a sample; the modal missing indication vector is used to indicate that a first v modal of the sample is missing or visible;​ initialization N The latent representation of samples is used V A neural network maps the latent representation of each sample to the feature space of the corresponding modality, obtains the reconstructed data representation of each modality of the sample, obtains the missing modality of each sample according to the modality missing indicator vector, uses the reconstructed data representation of the missing modality to complete the missing modality data, and obtains the complete multimodal data based on the visible modality data of each sample and the completed missing modality data; kernel alignment is performed on linear kernel matrices of each modality data and each sample hidden representation in complete multi-modal data, kernel alignment losses of each modality are weighted and fused to obtain a data alignment loss; a data reconstruction loss is constructed according to distances between each reconstructed data representation and corresponding modality data, and a total loss function is obtained according to the data reconstruction loss and the data alignment loss; the total loss function is optimized, clustering is performed by using optimized hidden representations of each sample, and data labels of each classification are obtained; the data reconstruction loss constructed according to distances between each reconstructed data representation and corresponding modality data comprises: the data reconstruction loss constructed according to distances between each reconstructed data representation and corresponding modality data is: ； wherein, is a data reconstruction loss, is a number of samples, is a number of modalities, is a number of visible modalities for the th sample, is a modality missing indicator vector, is the th modality data for the th sample, is a reconstructed data representation of the th modality data for the th sample, is a Euclidean distance; the kernel alignment performed on linear kernel matrices of each modality data and each sample hidden representation in complete multi-modal data comprises: a linear kernel matrix of each sample hidden representation in complete multi-modal data is calculated to obtain a hidden representation kernel matrix; a linear kernel matrix of each modality in complete multi-modal data is calculated to obtain a modality kernel matrix corresponding to each modality; kernel matrix alignment is performed by maximizing similarity of the hidden representation kernel matrix and the modality kernel matrix; the reconstructed data representation is: ； wherein, is a reconstruction data representation of the th modality for the th sample, is a latent representation of the th sample, denotes parameters of the th neural network.

2. The method of claim 1, wherein, the data alignment loss obtained by weighting and fusing kernel alignment losses of each modality comprises: the data alignment loss obtained by weighting and fusing kernel alignment losses of each modality is: ； wherein, is a data alignment loss, is a weight for the th modality, is a kernel alignment loss for the th modality.

3. The method of claim 1, wherein, optimization of the total loss function comprises: the data reconstruction loss and the data alignment loss are optimized to obtain optimized neural network parameters, hidden representations of each sample and weight parameters of each modality.

4. The method of claim 1, wherein, the total loss function is: ； wherein, is the total loss function, is the data reconstruction loss, is the data alignment loss, is a hyperparameter.

5. An apparatus for implicit representation based nonholonomic multimodal deep clustering, implemented by the method of any one of claims 1-4. the device comprises: The data acquisition module is configured to acquire multi-modal data, a modal missing indication vector, and a classification category number, wherein the multi-modal data comprises N modal data of the sample V The modal missing indication vector is used to indicate that the i-th modal of the sample is missing or visible. v modal data of the sample The data reconstruction module is configured to initialize N hidden representations of the samples, respectively map the hidden representations of the samples to feature spaces of corresponding modalities by using V the neural networks to obtain reconstructed data representations of the samples in the modalities, acquire missing modalities of the samples according to the modality missing indication vectors, complete missing modality data by using the reconstructed data representations of the missing modalities, and obtain complete multi-modality data according to the visible modality data of the samples and the completed missing modality data. a data alignment module configured to perform kernel alignment on linear kernel matrices of each modality data and each sample hidden representation in complete multi-modal data, and to obtain a data alignment loss by weighting and fusing kernel alignment losses of each modality; a loss construction module configured to construct a data reconstruction loss according to distances between each reconstructed data representation and corresponding modality data, and to obtain a total loss function according to the data reconstruction loss and the data alignment loss; a result output module configured to optimize the total loss function, and to obtain data labels of each classification by clustering by using optimized hidden representations of each sample. 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-5 when the computer program is executed by the processor. The processor implements the steps of the method in any one of claims 1 to 4 when executing the computer program.

Images