Multi-view representation learning method, clustering method, equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a multi-view representation learning method, a clustering method, equipment and a storage medium. The multi-view representation learning method comprises the following steps: inputting a sample multi-view data set into a pre-constructed multi-variation auto-encoder model to obtain shared variable information and specific variable information of each view, modeling and sampling specific variables by adopting a vMF distribution algorithm to obtain specific features of each view, and performing multi-view representation learning on the specific features of each view; after the shared variable information of all the views is fused, modeling and sampling are carried out on shared variables by adopting a Gumbel-Softmax distribution algorithm to obtain shared features, and the unique features and the shared features of each view are fused to obtain potential representation of each view; and performing reconstruction and iterative optimization on the potential representation of each view to obtain an optimized multi-variation auto-encoder model. According to the method provided by the invention, the limitation caused by Gaussian distribution is effectively solved, and a better clustering result can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-view representation learning method, a clustering method, a device and a storage medium. BACKGROUND

[0002] In recent years, multi-view clustering (MVC) methods have received extensive attention due to their ability to integrate complementary information from multiple sources or perspectives, providing a more comprehensive approach to mining target features and connections between different perspectives to understand complex data.

[0003] However, although MVC methods perform well in integrating multi-source information, existing methods have obvious deficiencies when dealing with complex data. Specifically, most methods rely on the assumption of Gaussian distribution in the latent space, which may cause problems such as collapse of the latent space when dealing with high-dimensional complex data or non-Gaussian distribution data, thereby hindering data alignment and information fusion and impairing similarity measurement between multi-views. For example, in MVC methods based on the Variational Autoencoder (VAE) framework, excessive reliance on the KL (Kullback-Leibler) divergence regularization term may cause the latent representation to be too concentrated, and sometimes even cause the latent space to collapse. This phenomenon occurs when the latent features of different perspectives become highly similar or overlapping, thereby weakening the effectiveness and reliability of the clustering results of the VAE-based MVC method. SUMMARY

[0004] Embodiments of the present application aim to provide a multi-view representation learning method, a clustering method, a device and a storage medium to solve the problem that the MVC method in the prior art may cause the latent space to collapse when dealing with high-dimensional complex data or non-Gaussian distribution data.

[0005] To solve the above technical problems, the embodiments of the present application provide the following technical solutions: According to a first aspect of the present application, a multi-view representation learning method is provided, the method comprising: obtaining a sample multi-view data set, the sample multi-view data set comprising a plurality of sample data, each sample data comprising a plurality of views; The multi-variational autoencoder model is constructed, the encoder of the multi-variational autoencoder model is used for latent space mapping and decoupling learning of each view in sample data, shared variable information and unique variable information of each view of the sample data are obtained, the vMF distribution algorithm is used for modeling and sampling of the unique variable based on the unique variable information of each view of the sample data, unique features of each view of the sample data are obtained, the shared variable information of all views in the sample data is fused, the Gumbel-Softmax distribution algorithm is used for modeling and sampling of the shared variable, shared features of the sample data are obtained, and the unique features of each view of the sample data and the shared features of the sample data are spliced and fused to obtain the latent representation of each view of the sample data. The decoder of the multi-variational autoencoder model is used for reconstructing the latent representation of each view of the sample data, and the multi-variational autoencoder model is jointly optimized based on a reconstruction loss and a KL divergence, so as to obtain an optimized multi-variational autoencoder model.

[0006] Optionally, the multi-variational autoencoder model comprises a plurality of variational autoencoders, and each variational autoencoder corresponds to one view of sample data.

[0007] Optionally, the encoder of the multi-variational autoencoder model is used for latent space mapping and decoupling learning of each view in sample data, shared variable information and unique variable information of each view of the sample data are obtained, and the method comprises the following steps. Based on each view of the sample data, latent space mapping is performed by one variational autoencoder in the multi-variational autoencoder model to obtain hidden variables of each view of the sample data. The hidden variables of each view of the sample data are input into a shared learning branch and a unique learning branch for decoupling learning, and shared variable information and unique variable information of each view of the sample data are obtained.

[0008] Optionally, the unique variable information comprises a unique mean vector and a unique concentration parameter, and the hidden variables of each view of the sample data are input into the unique learning branch for decoupling learning to obtain the unique variable information of each view of the sample data. The hidden variables of each view of the sample data are input into a neural network of the unique learning branch to obtain a unique hidden vector, the unique hidden vector is subjected to L2 norm normalization processing to obtain the unique mean vector, and the unique hidden vector is activated by an activation function to obtain the unique concentration parameter.

[0009] Optionally, the formula for modeling the unique variable by using the vMF distribution algorithm is as follows:

[0010] wherein, is a view-specific variable, is a view-specific variable, is a view-specific mean vector, is a view-specific mean vector, is a view-specific concentration parameter, is a view-specific concentration parameter, is a dimension of the view-specific variable, is a normalization constant for the view, is a normalization constant for the view.

[0011] Optionally, the shared variable information is a shared hidden vector, and the fusing of the shared variable information of all views in the sample data adopts a Gumbel-Softmax distribution algorithm to model and sample the shared variable, to obtain the shared feature of the sample data, including: splicing and fusing the shared hidden vectors of all views in the sample data to obtain a probability vector of the shared variable; based on the probability vector, using a Gumbel-Softmax distribution algorithm to model the shared variable, and sampling an approximately one-hot vector as the shared feature of the sample data.

[0012] Optionally, the KL divergence includes a first KL divergence corresponding to the view-specific variable and a second KL divergence corresponding to the shared variable, the prior distribution of the first KL divergence is a hyperspherical uniform distribution, and the posterior distribution is a vMF distribution, the prior distribution of the second KL divergence is a categorical uniform distribution, and the posterior distribution is a Softmax probability distribution.

[0013] According to a second aspect of the present application, a multi-view image clustering method is provided, the method comprising: based on the multi-view representation learning method of any one of the above, obtaining an optimized multi-varitional autoencoder model; obtaining a multi-view data set to be clustered, the multi-view data set to be clustered including a plurality of data to be clustered, each data to be clustered including a plurality of views; inputting the multi-view data set to be clustered into the optimized multi-varitional autoencoder model to obtain the latent representation of each view of the data to be clustered; fusing the latent representation of each view of the data to be clustered to obtain a fused latent representation of the data to be clustered; based on the fused latent representation of each data to be clustered in the multi-view data set to be clustered, clustering to obtain a clustering result of the multi-view data set to be clustered.

[0014] According to a third aspect of the present application, an electronic device is provided, comprising at least one processor and a memory connected to the at least one processor in communication, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0015] According to a fourth aspect of the present application, a computer storage medium is provided, the computer storage medium storing instructions or programs, when the instructions or programs are executed by at least one processor, enabling the at least one processor to perform the method described above.

[0016] The beneficial effects of the embodiments of the present application are: different from the prior art, in the embodiments of the present application, a multi-view representation learning method is provided, a multi-variate autoencoder model is constructed, the encoder of the multi-variate autoencoder model first maps and decouples the learning of each view in the sample data to obtain the shared variable information and the unique variable information of each view of the sample data; then, based on the unique variable information of each view, the vMF distribution algorithm is used to model and sample the unique variable to obtain the unique feature of each view of the sample data, and the shared variable information of all views of the sample data is fused, the Gumbel-Softmax distribution algorithm is used to model and sample the shared variable to obtain the shared feature of the sample data; then, the unique feature of each view of the sample data and the shared feature of the sample data are spliced and fused to obtain the latent representation of each view of the sample data; finally, the decoder of the multi-variate autoencoder model is used to reconstruct the latent representation of each view of the sample data, and the multi-variate autoencoder model is jointly optimized based on the reconstruction loss and the KL divergence to obtain the optimized multi-variate autoencoder model. The method of the present application fits the continuous unique variable to the hypersphere by using the vMF distribution, effectively solves the limitation of Gaussian distribution, and thus has more advantages than the method based on Gaussian assumption in processing the representation of complex data in multi-view clustering tasks, and further obtains better clustering results. BRIEF DESCRIPTION OF DRAWINGS

[0017] One or more embodiments are illustrated by way of example in the drawings, which are not intended to be limiting of the embodiments, and which will be described in the following. Elements having the same reference numbers in figures mean the same element. The drawings are not to scale.

[0018] Figure 1 is an architecture diagram of a multi-VAE model provided by the embodiments of the present application; Figure 2 is a flowchart of a multi-view representation learning method provided by the embodiments of the present application; Figure 3is a flowchart of a multi-view image clustering method provided by an embodiment of the present application. Figure 4 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0020] In addition, the technical features involved in each of the embodiments of the present application described below can be combined with each other as long as there is no conflict.

[0021] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0022] The proprietary technical terms involved in the present application are explained as follows: 1. Variational Autoencoder (VAE) is a generative model combining deep learning and probabilistic graphical models, which can learn low-dimensional latent variable representations of data and generate new data samples. Its core idea is to approximate the true posterior distribution through variational inference, realizing efficient data modeling and generation.

[0023] 2. Multi-view Clustering (MVC) is a method of clustering analysis using multiple feature sets (perspectives) of data objects, suitable for data of the same object in different dimensions or sources (such as visual features and text descriptions of an image). Its core idea is to integrate multi-view information to improve clustering performance.

[0024] 3. Multi-view Representation Learning aims to extract unified or complementary feature representations from multiple data sources (perspectives) to improve the performance of downstream tasks (such as clustering and classification). Its core idea is to utilize the consistency and complementarity between different perspectives to learn more robust and discriminative low-dimensional representations.

[0025] 4.Multi-view Datasets, refers to a dataset containing multiple different perspectives, different features or different modal data of the same object or scene.

[0026] 5.KL (Kullback-Leibler) divergence, also known as relative entropy, is an asymmetric measure of the difference between two probability distributions P and Q.

[0027] 6.von Mises-Fisher (vMF) distribution, is a probability distribution defined on the unit hypersphere, suitable for describing directional data.

[0028] 7.Gumbel-Softmax distribution (also known as Concrete distribution), is a continuous probability distribution used for differentiable sampling of discrete distributions.

[0029] 8.Softplus activation function, is a smooth, monotonically increasing activation function, which is a smooth approximation of ReLU, with the property of being differentiable everywhere.

[0030] 9.Hypersphere, is a sphere in high-dimensional space, defined as the set of all points with equal distance from the center point.

[0031] 10.one-hot vector, is a vector containing K categories of discrete variables, and only one element is 1 (corresponding to the category position), and the rest is 0.

[0032] The present application aims to address the limitations of traditional multi-view clustering methods in handling high-dimensional data or non-Gaussian distribution data, and proposes an innovative multi-view representation learning method and clustering method. Through vMF distribution, the multi-view image data is efficiently decoupled, and the unique features are embedded into the hypersphere space to capture the complex information that may be ignored by traditional Gaussian-based methods. The method of the present application uses a multi-VAE architecture, which inputs each view of a multi-view output into a VAE, normalizes the L2 norm, and extracts the distribution of the latent space. Then, the vMF distribution is used to model the unique variables extracted, arrange the data points on the hypersphere, and promote the close distribution of data of the same class through the reconstruction loss and KL divergence of the lower VAE, thereby achieving better clustering performance.

[0033] Please refer to Figure 1 , Figure 1 is the architecture diagram of the multi-VAE model provided by the embodiments of the present application, which includes multiple VAEs, each VAE corresponding to a view of the multi-view data. In an embodiment, each VAE includes the same encoder and decoder. As Figure 1 shown, the multi-view dataset Medium sample data Different views are input into each VAE of the multi-VAE model, and the encoder of each VAE obtains the unique features of each view. and shared features And extract the shared features from all views Merge and stitch them together to form a complete view sharing feature Then, based on each view, share features between views. With the unique features of this view The potential representation of the view is obtained by stitching the images together. ], based on the potential representation of this view [ The reconstructed image is generated through the VAE decoder.

[0034] Please refer to Figure 2 , Figure 2 This is a flowchart of a multi-view representation learning method provided in an embodiment of this application. The method specifically includes the following steps: Step S201: Obtain a sample multi-view dataset, which includes multiple sample data, and each sample data includes multiple views.

[0035] A multi-view dataset refers to a dataset containing data on the same object or scene from multiple different perspectives, with different features, or different modalities. For example, in computer vision, a multi-view dataset might consist of images of several objects at different rotation angles. In face recognition, it could be facial images of several humans under different poses, lighting conditions, and expressions. In autonomous driving, a multi-view dataset can also be data (images, point clouds, geolocation) collected by different sensors (vehicle cameras, LiDAR, GPS, etc.). Each piece of multi-view data includes multiple views.

[0036] In this embodiment, the multi-view dataset is image data captured from multiple different perspectives based on objects of different categories. For example, the multi-view dataset is image data captured from V perspectives of objects of K categories.

[0037] In some embodiments, the multi-view dataset is further used to clean and preprocess the multi-view dataset according to its characteristics, including steps such as deduplication, handling missing values, and standardization, so as to adjust all views to the same size range to eliminate size inconsistencies caused by differences in shooting angle, distance, or resolution.

[0038] In step S202, a multi-variance auto-encoder model is constructed. The encoder of the multi-variance auto-encoder model is used to perform latent space mapping and decoupling learning on each view of the sample data, to obtain shared variable information and unique variable information of each view of the sample data. Based on the unique variable information of each view of the sample data, a vMF distribution algorithm is used to model and sample the unique variable, to obtain unique features of each view of the sample data. After the shared variable information of all views of the sample data is fused, a Gumbel-Softmax distribution algorithm is used to model and sample the shared variable, to obtain shared features of the sample data. The unique features of each view of the sample data and the shared features of the sample data are spliced and fused, to obtain latent representations of each view of the sample data.

[0039] In an embodiment, the architecture of the multi-VAE model is as shown in FIG. 2. The multi-VAE model includes multiple VAEs, each of which has the same encoder and decoder, and each of which processes one view of the sample data. Specifically, each view of the sample data in the sample multi-view data set is input into the encoder of one VAE of the multi-VAE model for latent space mapping, to obtain hidden variables of each view of the sample data. Each view of the image of each category contains both shared information of the object of the category and unique information of the object of the category. To obtain the information, a shared learning branch and a unique learning branch are provided in the encoder. The hidden variables of each view of the sample data are input into the shared learning branch and the unique learning branch for decoupling learning, to obtain shared variable information and unique variable information of each view of the sample data, respectively. Figure 1 In this application, the shared variable is discrete category information, and the unique variable is continuous unique feature information. For the unique variable, a vMF distribution algorithm is used for modeling and sampling. For the shared variable, a Gumbel-Softmax distribution algorithm is used for modeling and sampling.

[0040] To implement the vMF distribution algorithm for modeling the unique variable, the unique variable information output by the encoder includes vMF distribution parameters of the unique variable, including a unique mean vector and a unique concentration parameter. Specifically, the hidden variables of each view of the sample data are input into the neural network of the unique learning branch for learning, to obtain unique hidden vectors. The unique hidden vectors are subjected to L2 norm normalization processing to obtain the unique mean vector, and the unique hidden vectors are activated by an activation function (such as a softplus activation function) to obtain the unique concentration parameter. Further, the unique variable is modeled according to the vMF distribution parameters output by the encoder, using the following formula:

[0041]

[0042] ​wherein, is the unique variable of the th view, is the unique mean vector of the th view, is the unique concentration parameter of the th view, is the dimension of the unique variable, is the normalization constant of the th view.

[0043] To realize the modeling of the shared variable by using the Gumbel-Softmax distribution algorithm, the encoder of all views first splices and fuses the shared variable information (i.e., the shared hidden vector) of all views in the sample data, and outputs the Gumbel-Softmax distribution parameter based on the fused view shared vector, that is, the probability vector of the shared variable. Then, based on the probability vector of the shared variable, the Gumbel-Softmax distribution algorithm is used to model the shared variable, and an approximately one-hot vector is sampled as the shared feature of the sample data. The core method of modeling the shared variable by using the Gumbel-Softmax distribution algorithm is to add Gumbel noise to each category weight in the probability vector to obtain:

[0044] wherein, , is the temperature coefficient. Then, a continuous shared vector is obtained through a layer of softmax reparameterization.

[0045] wherein, the lower the temperature coefficient , the closer to the real one-hot. The input probability vector is generated by the encoders of all views. The output is a continuous one-hot approximation . . Wherein, N is the sample number, K is the category number, is the category serial number.

[0046] The unique variable is modeled and sampled by using the vMF distribution to obtain the unique feature of the sample data, the shared variable is modeled and sampled by using the Gumbel-Softmax distribution algorithm to obtain the shared feature of the sample data, and the unique feature of the sample data in each view and the shared feature of the sample data are fused to obtain the final feature​ By stitching and merging the data, we can obtain the latent representation of the sample data in each view. ].

[0047] Step S203: The latent representation of the sample data in each view is reconstructed by the decoder of the multivariate autoencoder model, and the multivariate autoencoder model is jointly optimized based on the reconstruction loss and KL divergence to obtain the optimized multivariate autoencoder model.

[0048] In this application, two decoupled representations learned from the latent space are ultimately merged into a single latent representation. Specifically, the unique and shared variables of each view are combined into a new variable. This new variable is decoded and reconstructed to obtain reconstructed data, based on which the reconstruction loss can be calculated. In one embodiment, the reconstruction loss can be expressed as:

[0049] Since this application decouples multi-view data into two variables—shared variables and unique variables—it is necessary to consider the KL divergence of these two variables separately. The formula for calculating the first KL divergence corresponding to the unique variable is as follows:

[0050] In one embodiment, the prior distribution of the first KL divergence For a hypersphere uniform distribution, the posterior distribution is... If the distribution is vMF, then the formula for calculating the first KL divergence can be expressed as:

[0051] in, It is the gamma function. It is a modified Bessel function of the first kind. It is the first The unique concentration parameter of each view. It is a dimension of unique variables. It is the first The normalization constant of each view.

[0052] The formula for calculating the second KL divergence corresponding to the shared variable is:

[0053] In one embodiment, the prior distribution of the second KL divergence For uniform distribution of classification, posterior distribution If the probability distribution is Softmax, then the formula for calculating the second KL divergence can be expressed as follows:

[0054] wherein, is the number of categories of shared variables.

[0055] Based on this, the ELBO (Evidence Lower Bound) optimization target of the multi-variational autoencoder model is: = - -

[0056] wherein, and are weight coefficients. By minimizing to minimize the reconstruction error and minimize the KL divergence, the multi-variational autoencoder model is jointly optimized to obtain the optimized multi-variational autoencoder model. In an embodiment, when the number of iterations reaches the maximum number of iterations, the optimization of the multi-variational autoencoder model is stopped, and the optimized multi-variational autoencoder model is obtained.

[0057] Please refer to Figure 3 , Figure 3 is a flowchart of a multi-view image clustering method provided by an embodiment of the present application, which specifically includes the following steps: Step S301, an optimized multi-variational autoencoder model is obtained by using the multi-view representation learning method in Figure 2 .

[0058] Step S302, a multi-view data set to be clustered is obtained, and the multi-view data set to be clustered includes a plurality of data to be clustered, and each data to be clustered includes a plurality of views.

[0059] Wherein, the multi-view data set to be clustered is the image data captured by V views of one or more categories of objects in the aforementioned K categories of objects. In an embodiment, the multi-view data set to be clustered can be the aforementioned sample multi-view data set or a new multi-view data set.

[0060] Step S303, the multi-view data set to be clustered is input into the optimized multi-variational autoencoder model to obtain the latent representation of each view of the data to be clustered.

[0061] Step S304, the latent representation of each view of the data to be clustered is fused to obtain the fused latent representation of the data to be clustered.

[0062] Step S305, clustering is performed based on the fused latent representation of each data to be clustered in the multi-view data set to be clustered to obtain the clustering result of the multi-view data set to be clustered.

[0063] In an embodiment, the fused latent representation of each to-be-clustered data in the to-be-clustered multi-view data set is input into a von Mises-Fisher Mixture Model (vMFMM) to obtain a clustering result of the to-be-clustered multi-view data set.

[0064] The multi-view clustering method based on the VAE architecture and using the vMF distribution proposed in the application is compared with the classic clustering algorithm K-means and the multi-view clustering method based on the Gaussian assumption (referred to as "Multi-VAE") in terms of the clustering effect based on the NMI (Normalized Mutual Information) value, as shown in Table 1 below. Among them, the three methods are repeated 10 times, and the average value is taken as the comparison index.

[0065] Table 1

[0066] As can be seen from the comparison results in Table 1, the method of the application can obtain better multi-view clustering results (higher NMI value) compared with the classic clustering algorithm K-means and the multi-view clustering method based on the Gaussian assumption.

[0067] The multi-view representation learning method provided in the embodiments of the application constructs a multi-variate autoencoder model, and the encoder of the multi-variate autoencoder model first performs latent space mapping and decoupling learning on each view in the sample data to obtain shared variable information and unique variable information of each view of the sample data; then, based on the unique variable information of each view, the vMF distribution algorithm is used to model and sample the unique variable to obtain unique features of each view of the sample data, and the shared variable information of all views of the sample data is fused, and the Gumbel-Softmax distribution algorithm is used to model and sample the shared variable to obtain shared features of the sample data; then, the unique features of each view of the sample data and the shared features of the sample data are spliced and fused to obtain the latent representation of each view of the sample data; finally, the decoder of the multi-variate autoencoder model is used to reconstruct the latent representation of each view of the sample data, and the multi-variate autoencoder model is jointly optimized based on the reconstruction loss and the KL divergence to obtain an optimized multi-variate autoencoder model. The method of the application fits the continuous unique variable to the hypersphere using the vMF distribution, effectively solving the limitations brought by the Gaussian distribution, so that the method based on the Gaussian assumption has more advantages in processing the representation of complex data in the multi-view clustering task, and thus a better clustering result is obtained.

[0068] According to the embodiments of the application, an electronic device is provided, such as Figure 4A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1. The electronic device 100 can include a processor 10, a communication interface 30, a memory 20, and a communication bus. The processor 10, the communication interface 30, and the memory 20 can communicate with each other through the communication bus. The processor 10 can invoke a logical instruction in the memory 20 to execute the multi-view representation learning method or the multi-view image clustering method described above.

[0069] In addition, the logical instruction in the memory 20 described above can be implemented in the form of a software function unit and sold or used as an independent product. In this case, the logical instruction can be stored in several computer-readable storage media. Based on this understanding, the technical solution of the present application or the part of the technical solution that essentially contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the multi-view representation learning method or the multi-view image clustering method described above. The storage medium described above includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, and various other media that can store program codes.

[0070] According to an embodiment of the present application, a computer-readable storage medium is provided. The computer-readable storage medium is of the type described above and stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the multi-view representation learning method or the multi-view image clustering method described above.

[0071] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the technical solution described above or the part of the technical solution that essentially contributes to the related art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment or some part of the embodiment.

[0072] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application. Therefore, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-view representation learning method, characterized in that, The method includes: Obtain a sample multi-view dataset, which includes multiple sample data, and each sample data includes multiple views; A multivariate autoencoder model is constructed. The encoder of the multivariate autoencoder model performs latent space mapping and decoupling learning on each view in the sample data to obtain the shared variable information and unique variable information of the sample data in each view. Based on the unique variable information of the sample data in each view, the vMF distribution algorithm is used to model and sample the unique variables to obtain the unique features of the sample data in each view. The shared variable information of all views in the sample data is fused, and the Gumbel-Softmax distribution algorithm is used to model and sample the shared variables to obtain the shared features of the sample data. The unique features of the sample data in each view and the shared features of the sample data are concatenated and fused to obtain the latent representation of the sample data in each view. The decoder of the multivariate autoencoder model reconstructs the latent representation of the sample data in each view, and the multivariate autoencoder model is jointly optimized based on the reconstruction loss and KL divergence to obtain the optimized multivariate autoencoder model.

2. The method according to claim 1, characterized in that, The multivariate autoencoder model includes multiple variational autoencoders, each of which corresponds to a view of the sample data.

3. The method according to claim 2, characterized in that, The process of using the encoder of the multivariate autoencoder model to perform latent space mapping and decoupling learning on each view of the sample data to obtain shared variable information and unique variable information of the sample data in each view includes: Based on each view in the sample data, the latent space is mapped by a variational autoencoder in the multivariate autoencoder model to obtain the hidden variables of the sample data in each view. The hidden variables of the sample data in each view are input into the shared learning branch and the specific learning branch for decoupled learning, so as to obtain the shared variable information and specific variable information of the sample data in each view respectively.

4. The method according to claim 3, characterized in that, The unique variable information includes a unique mean vector and a unique concentration parameter. The process of inputting the hidden variables of the sample data in each view into a unique learning branch for decoupled learning, to obtain the unique variable information of the sample data in each view, includes: The hidden variables of the sample data in each view are input into the neural network of the unique learning branch to obtain the unique hidden vector. The unique hidden vector is normalized by L2 norm to obtain the unique mean vector. The unique hidden vector is activated by an activation function to obtain the unique concentration parameter.

5. The method according to claim 4, characterized in that, The formula for modeling the specific variables using the vMF distribution algorithm is as follows: in, It is the first Variables specific to each view It is the first The unique mean vector of each view It is the first The unique concentration parameter of each view. It is a dimension of unique variables. It is the first The normalization constant of each view.

6. The method according to any one of claims 1 to 5, characterized in that, The shared variable information is a shared hidden vector. The process involves fusing the shared variable information from all views in the sample data, and using the Gumbel-Softmax distribution algorithm to model and sample the shared variables to obtain the shared features of the sample data, including: After concatenating and fusing the shared hidden vectors of all views in the sample data, the probability vector of the shared variable is obtained. Based on the probability vector, the Gumbel-Softmax distribution algorithm is used to model the shared variables, and an approximately one-hot vector is sampled as the shared feature of the sample data.

7. The method according to any one of claims 1 to 5, characterized in that, The KL divergence includes a first KL divergence corresponding to the specific variables and a second KL divergence corresponding to the shared variables. The prior distribution of the first KL divergence is a hypersphere uniform distribution, and the posterior distribution is a vMF distribution. The prior distribution of the second KL divergence is a classification uniform distribution, and the posterior distribution is a Softmax probability distribution.

8. A multi-view image clustering method, characterized in that, The method includes: Based on the multi-view representation learning method according to any one of claims 1 to 7, an optimized multivariate autoencoder model is obtained; Obtain a multi-view dataset to be clustered, wherein the multi-view dataset to be clustered includes multiple datasets to be clustered, and each dataset to be clustered includes multiple views; The multi-view dataset to be clustered is input into the optimized multivariate autoencoder model to obtain the latent representation of the data to be clustered in each view. The latent representations of the data to be clustered in each view are fused to obtain the fused latent representation of the data to be clustered. Clustering is performed based on the fusion latent representation of each data in the multi-view dataset to be clustered, and the clustering result of the multi-view dataset to be clustered is obtained.

9. An electronic device, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores instructions or programs that, when executed by at least one processor, cause the at least one processor to perform the method as described in any one of claims 1 to 8.

Images