An incomplete multi-view clustering system, method and storage medium based on variational autoencoder

Abstract

The present invention provides an incomplete multi-view clustering system, method and storage medium based on a variational autoencoder. The incomplete multi-view clustering system includes a deep multi-view probabilistic coding network module, a multi-view feature fusion module, a feature consistency maintenance module, a cluster allocation module and a deep multi-view probabilistic decoding network module. The beneficial effects of the present invention are: 1) The present invention applies a probabilistic encoder to an incomplete multi-view clustering task for the first time, and can flexibly and efficiently handle clustering tasks with arbitrary missing view data without the need for additional data completion steps. The complexity of the proposed model is linearly related to the number of viewpoints and can be applied to scenes with multiple viewpoints; 2) The present invention adopts a distribution-based clustering prediction method. In the proposed model, cluster prediction results can be directly obtained without executing additional clustering algorithms on the extracted features to obtain the final results.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition, and in particular to an incomplete multi-view clustering system, method and storage medium based on a variational autoencoder. Background Art

[0002] In recent years, multi-view clustering has garnered widespread attention in the fields of machine learning and computer vision. Existing multi-view clustering methods generally rely on a core assumption: all multi-view samples have complete view data. However, in real-world applications, multi-view samples often suffer from missing view data due to factors such as sensor failure and information omissions. To address this issue, numerous incomplete multi-view clustering methods have been proposed. These methods can be categorized into completion-based methods and non-completion-based methods, depending on how they handle missing view data.

[0003] Traditional completion-based methods, including matrix factorization, multi-kernel learning, and graph learning, are commonly used. These methods have been recognized for their theoretical interpretability and physical significance. However, since most of these methods are based on shallow models, their clustering performance is relatively limited due to their limited feature extraction capabilities. Furthermore, these methods often come with high computational complexity and have difficulty handling mixed data types such as text, audio, and video. With the advancement of deep learning technology, completion-based deep incomplete multi-view clustering methods have gradually emerged, such as Dual Contrastive Prediction for Incomplete Multi-view Representation Learning (DCP). These methods leverage the powerful feature extraction and data generation capabilities of deep neural networks to achieve excellent performance in completing missing views and achieving clustering performance. However, the effectiveness of these methods is highly dependent on the quality of the completed views, and rigorously proving the accuracy of the completion in theory remains a challenge. Therefore, the process of completing missing views may introduce additional noise, which can degrade clustering performance.

[0004] Compared to completion-based methods, non-completion-based methods directly utilize information from the remaining viewpoints, avoiding the noise that may be introduced by the completion process. The main goal of these methods is to transform the original incomplete multi-view data into a shared space. In this space, they fuse information from all available viewpoints to obtain a consistent representation, thereby effectively mining clustering information. The key to these methods lies in how to effectively fuse information from different viewpoints. Common fusion strategies include concatenation and weighted summation of viewpoint features. For example, Deep Incomplete Multi-View Clustering via Mining Cluster Complementarity (DIMVC) uses feature concatenation to project the fused features into a high-dimensional space to obtain features that are more conducive to clustering. However, these fusion strategies face the challenge of capturing the complex nonlinear relationships between different views and may overlook the inherent information differences between views, which can lead to information imbalance in the fused features. In summary, existing non-completion-based methods often struggle to achieve optimal clustering results. Summary of the Invention

[0005] In order to solve the problems in the prior art, the present invention provides an incomplete multi-view clustering system, method and storage medium based on variational autoencoder.

[0006] The present invention provides an incomplete multi-view clustering system based on a variational autoencoder, comprising a deep multi-view probabilistic coding network module, a multi-view feature fusion module, a feature consistency maintenance module, a cluster assignment module and a deep multi-view probabilistic decoding network module, wherein:

[0007] The deep multi-view probabilistic coding network module maps the original high-dimensional data encoding to a low-dimensional feature space and obtains the feature distribution of the data, thereby extracting condensed information from the original multi-view data with redundant information;

[0008] The multi-view feature fusion module treats data features from different viewpoints as viewpoint experts and uses an expert product method to effectively fuse data features from all viewpoints when any viewpoint data is missing, thereby obtaining fused features that are unanimously recognized by experts from different viewpoints and contain shared information between multiple viewpoints.

[0009] The feature consistency maintenance module reduces the distribution difference between the fused features and the features of each view, and alleviates the information imbalance problem that may exist in the fused features when some view data is missing;

[0010] The cluster assignment module: infers cluster assignment by fusing features, generates category pseudo labels during model training to guide the model to learn features with a good cluster structure, and directly obtains clustering results through the cluster assignment module after model training is completed;

[0011] The deep multi-view probabilistic decoding network module: the decoding network generates multi-view reconstructed data based on the fusion feature decoding, and minimizes the reconstruction loss between the original data and the reconstructed data so that the fusion feature retains more information;

[0012] The data processed by the incomplete multi-view clustering system includes image data, audio data, and text data.

[0013] As a further improvement of the present invention, the deep multi-view probability coding network module is composed of a view-specific probability encoder corresponding to each view, and the probability encoder structure under different view angles can be expanded according to the data input type of the corresponding view angle.

[0014] As a further improvement of the present invention, the view-specific probability encoder is constructed based on a fully connected linear layer. Each probability encoder consists of four layers. Each of the first three layers consists of a linear layer, and the fourth layer consists of two independent linear layers with the same input and output dimensions. The two linear layers of the fourth layer encode the mean and standard deviation of the feature distribution respectively. The linear layer encoding the standard deviation is followed by a Softplus activation function to match the non-negative characteristic of the standard deviation. In each probability encoder, the two adjacent layers are connected in series by a linear rectification activation function, and the output dimensions of the linear layer are set to 500, 500, 2000, and d respectively, where d is the feature dimension of the desired data feature.

[0015] As a further improvement of the present invention, in the multi-view feature fusion module, the output obtained by the incomplete multi-view data through the deep multi-view probability coding network module is defined as Where N is the total number of samples of incomplete multi-view data, V represents the total number of viewpoints of the input data, The view feature obtained by encoding the vth view data of the i-th incomplete multi-view input sample corresponds to a probability distribution, μ represents its mean vector, σ represents the standard deviation vector corresponding to the diagonal variance of its covariance matrix, and the multi-view feature fusion module based on the view missing index information is used to fuse the view features of multiple views to obtain:

[0016]

[0017]

[0018] in, is the fusion feature of the i-th input sample, M is the perspective missing index matrix, if the v-th perspective data of the i-th input sample is missing, then M iv =0, otherwise M iv =1.

[0019] As a further improvement of the present invention, in the feature consistency maintenance module, the following feature consistency loss function based on the relative entropy between the corresponding distribution of the fusion feature and the corresponding distribution of the view feature is adopted:

[0020]

[0021] As a further improvement of the present invention, after obtaining the fused features through the multi-view feature fusion module, the cluster assignment module uses a resampling technique to sample a feature sample from the corresponding distribution of the fused features, calculates the probability of the sample under the prior feature distribution of different categories, and obtains an approximate clustering probability by performing a normalization operation on the probabilities under all categories;

[0022] In the cluster assignment module, in order to promote the model to learn fusion features with good clustering structure, a feature clustering structure loss function is designed. The feature clustering structure loss function uses the approximate clustering probability as the weight and calculates the weighted relative entropy between the corresponding distribution of the fusion feature and the prior feature distribution of each category. It is specifically expressed as:

[0023]

[0024] in,

[0025]

[0026] γ ic Represents the feature sample z sampled from the corresponding distribution of the i-th fusion feature i The approximate probability of belonging to the cth category, ε i represents a sample randomly sampled from a multivariate Gaussian distribution with mean 0 and covariance matrix equal to the identity matrix, μ c represents the mean of the prior feature distribution of the cth category, σ c Represents the standard deviation of the prior feature distribution of the cth category, there are C {μ c ; σ c} corresponds to C categories of prior feature distribution, As a learnable parameter, it participates in the parameter update during model training. After the model training is completed, the cluster assignment of the i-th incomplete multi-view input sample is obtained by the following formula:

[0027]

[0028] As a further improvement of the present invention, the deep multi-view probabilistic decoding network module is composed of multiple independent view-specific probabilistic decoders, each of which uses feature samples sampled from the corresponding distribution of fused features. Decoding generates reconstructed data corresponding to the viewing angle, Represents a sample randomly sampled from a multivariate Gaussian distribution with a mean of 0 and a covariance matrix of the identity matrix, which promotes the fusion feature encoding of multi-view by minimizing the reconstruction loss between the original data and the reconstructed data at each viewpoint. Figure 1 consistent information that is conducive to clustering.

[0029] As a further improvement of the present invention, in the deep multi-view probabilistic decoding network module, the view-specific probabilistic decoder of each view is designed to have a structure opposite to that of the corresponding view probabilistic encoder. Each probabilistic decoder contains four linear layers, and two adjacent linear layers are connected in series by a linear rectification activation function. For the view-specific probabilistic decoder of the vth view, the output dimensions of each linear layer are set to 2000, 500, 500 and d respectively. v , where d v is the original data dimension of the v-th perspective;

[0030] In the deep multi-view probabilistic decoding network module, different incomplete multi-view data reconstruction loss functions are designed for different types of original data values. In the vth view, if the input original data value is a real number, the incomplete multi-view data reconstruction loss function is expressed as:

[0031]

[0032] in, Represents the original data of the vth perspective of the i-th input sample, Represents the reconstructed data generated by decoding the original data through the model; if the input original data value is binary type, the incomplete multi-view data reconstruction loss function is expressed as:

[0033]

[0034] The model training loss function of the incomplete multi-view clustering system is expressed as:

[0035]

[0036] Among them, α and β are adjustable hyperparameters.

[0037] The present invention also discloses an incomplete multi-view clustering method based on a variational autoencoder, which includes a model parameter initialization stage and a model training loss function optimization stage. The model parameter initialization stage includes the following steps:

[0038] The first step is initialization: first, fill the missing view data in the original incomplete multi-view data with 0 values, record all the single-view samples that are not missing under each view, the complete multi-view samples without missing view data, and the incomplete multi-view samples with missing view data, set the data iterator, load the required number of sample data in each iteration, and set the target number of clusters C;

[0039] The second step is to initialize the parameters of the encoding-decoding network: for each viewpoint, the probability encoder-probabilistic decoder uses all the non-missing single-view samples under the corresponding viewpoint, encodes the features through the probability encoder, and decodes the features with the probability decoder to generate reconstructed data, and calculates the mean square error loss between the original data and the reconstructed data. in represents the number of non-missing single-view samples at the v-th view. The training goal is to minimize the mean square error loss between the reconstructed data and the original data at each view. The parameters of the probability encoder-probability decoder are iteratively updated, and the number of training iterations is set as required. j represents the sample subscript;

[0040] The third step is to initialize the category prior feature distribution: a complete multi-view sample without missing view data is fed into the model, the data features of each view are encoded through the view-specific probability encoder, and the features of different viewpoints of the same sample are added and averaged to obtain Average features, N com Represents the number of complete multi-view samples without missing view data, for N com The K-means algorithm is executed on the average features, and the C cluster centers obtained after the execution are set as the mean of the category prior feature distribution and All are set to 1 to complete the model parameter initialization;

[0041] The model training loss function optimization stage includes:

[0042] Step 1: Set hyperparameters: Set the learning rate of the training process to the desired value, set the desired number of training rounds, and hyperparameters α = 1 and β = 10;

[0043] Step 2: Multi-view feature fusion: Based on the deep multi-view probability coding network module and the multi-view feature fusion module, the incomplete multi-view data is encoded by each view-specific probability encoder to obtain the corresponding view features. Then, according to formulas (1) and (2), the fusion feature is obtained by the expert product method.

[0044] Step 3: Multi-view data reconstruction: Using reparameterization techniques, sample from the corresponding distribution of its fusion features The feature samples obtained Input the deep multi-view probability decoding network module to decode and generate the reconstructed data of each view

[0045] Step 4: Generate pseudo labels for feature categories: Use reparameterization techniques to sample from the corresponding distribution of fusion features Using the obtained feature sample z i , calculate the approximate probability of belonging to each category according to formula (5)

[0046] Step 5: Model training loss calculation: According to L ch , L KL 、 Using formula (9), calculate the model training loss

[0047] Step 6: Gradient feedback optimization: Based on the model training loss obtained in step 5, use the stochastic gradient descent optimization algorithm to update all learnable parameters in the model.

[0048] Step 7, iterative training: according to the training rounds set in step 1, repeat the model training process from step 2 to step 6. After the iteration is completed, the final incomplete multi-view clustering model is obtained;

[0049] The data processed by the incomplete multi-view clustering method includes image data, audio data, and text data.

[0050] The present invention further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the method described in the present invention when called by a processor.

[0051] The beneficial effects of the present invention are: 1) The present invention applies the probabilistic encoder to the incomplete multi-view clustering task for the first time, and can flexibly and efficiently handle clustering tasks with arbitrary missing view data without the need for additional data completion steps. The complexity of the proposed model is linearly related to the number of viewpoints and can be applied to scenes with multiple viewpoints; 2) The present invention adopts a distribution-based clustering prediction method. The clustering prediction results can be directly obtained in the proposed model without the need to perform additional clustering algorithms on the extracted features to obtain the final results. Therefore, the present invention is more likely to obtain better clustering performance; 3) The experimental results of comparing the present invention with the most advanced completion-based and non-completion-free algorithms on four incomplete multi-view datasets show that the present invention can obtain the best clustering performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a structural diagram of the incomplete multi-view clustering system of the present invention;

[0053] Figure 2 is a schematic diagram of a view-specific probability encoder of the present invention;

[0054] Figure 3 Schematic diagram of the multi-view feature fusion module of the present invention;

[0055] Figure 4 Schematic diagram of the feature consistency maintenance module of the present invention;

[0056] Figure 5 This is a schematic diagram of the function of the feature consistency module of the present invention;

[0057] Figure 6 Schematic diagram of the perspective-specific probability decoder of the present invention. DETAILED DESCRIPTION

[0058] The present invention discloses an incomplete multi-view clustering system and method based on a variational autoencoder. It can comprehensively utilize deep learning technology, digital image processing technology, machine learning theory, etc. to realize unsupervised clustering of multi-view data. It can be applied to security scenarios such as face recognition and vehicle recognition, and can also be flexibly applied to disease diagnosis, image retrieval, recommendation system analysis, financial analysis, multimedia analysis and other fields.

[0059] The present invention discloses an incomplete multi-view clustering system based on variational autoencoder, which realizes more effective and flexible clustering of incomplete multi-view data.

[0060] like Figure 1 As shown in the figure, the incomplete multi-view clustering system based on variational autoencoder disclosed in the present invention mainly consists of five modules: deep multi-view probabilistic coding network module, multi-view feature fusion module, feature consistency maintenance module, cluster assignment module and deep multi-view probabilistic decoding network module.

[0061] Deep Multi-view Probabilistic Coding Network Module: This module extracts condensed information from redundant raw multi-view data by mapping the original high-dimensional data into a low-dimensional feature space and obtaining the data's characteristic distribution.

[0062] Multi-view feature fusion module: By treating data features from different viewpoints as viewpoint experts, the expert product method is used to obtain fusion features that are unanimously recognized by experts from different viewpoints. On the one hand, it can effectively and flexibly integrate the information of the remaining viewpoint data even when data from any viewpoint is missing. On the other hand, the fusion features obtained by the expert product method can contain more information shared by multiple viewpoints, which is conducive to exploring the clustering structure shared by different viewpoints of data.

[0063] Feature consistency maintenance module: By reducing the distribution difference between fused features and view features, it alleviates the information imbalance problem that may exist in fused features when some view data is missing, preventing information from some viewpoints from being ignored, which may lead to suboptimal clustering results. Cluster assignment module: Inferring cluster assignments by fused features. On the one hand, it generates category pseudo-labels during model training to guide the model to learn features with a good clustering structure. On the other hand, after model training is completed, clustering results can be directly obtained through this module, avoiding the need to execute additional clustering algorithms.

[0064] Deep multi-view probabilistic decoding network module: The decoding network generates multi-view reconstructed data based on fused feature decoding, and by minimizing the reconstruction loss between the original data and the reconstructed data, the fused features retain more information from each view. For example, in multi-sensor monitoring, different pedestrian objects can be distinguished. Even if the background changes drastically in different views, the clothing and appearance of the same pedestrian object remain unchanged in multiple views. In medical lesion detection, image data obtained by multiple detection methods are used as different views, where the same type of lesion has consistent pathological characteristics in multiple views.

[0065] Structure Introduction

[0066] Deep multi-view probabilistic coding network module: Existing incomplete multi-view clustering models mostly use matrix decomposition, autoencoders and other methods to obtain meaningful data features under unsupervised conditions. Among them, traditional methods using matrix decomposition, subspace learning and other technologies have weak feature extraction capabilities and limited flexibility when the view data is incomplete; while deep methods using deep autoencoders, deep encoders and the like can extract higher-order features from the data. Different from the deterministic encoder used in the autoencoder, the present invention uses a probabilistic encoder in a variational autoencoder. The encoded data features include the mean and standard deviation (variance) of its feature distribution, where the mean can be understood as the identification feature in the deterministic encoder, and the standard deviation represents the uncertainty of the identification feature. Therefore, compared to the deterministic encoder, the data features encoded by the probabilistic encoder can provide more information about the original data. For feature extraction of multi-view data, the present invention designs a deep multi-view probabilistic coding network module, which consists of a view-specific probabilistic encoder corresponding to each view. The probabilistic encoder structure under different viewpoints can be flexibly expanded according to the data input type of the corresponding viewpoint. Such as Figure 2 As shown, the present invention adopts a probability encoder constructed based on a fully connected linear layer. Each view-specific probability encoder consists of four layers. Each of the first three layers consists of a linear layer. The fourth layer is equipped with two independent linear layers with the same input and output dimensions. The two linear layers in the fourth layer encode the mean and standard deviation of the data feature distribution, respectively. The linear layer encoding the standard deviation is followed by a Softplus activation function to match the non-negative characteristic of the standard deviation. In each probability encoder, the two adjacent layers are connected in series by a linear rectifier activation function (ReLU). The output dimensions of the linear layer are set to 500, 500, 2000, and d, respectively, where d is the feature dimension of the desired data feature.

[0067] Multi-view feature fusion module: The output of the view-specific probability encoder under each view is defined as This output is the output obtained by the deep multi-view probabilistic coding network module of the incomplete multi-view data, where N is the total number of samples of the incomplete multi-view data, V is the total number of views of the input data, The perspective feature obtained by encoding the vth perspective data of the i-th incomplete multi-perspective input sample corresponds to a probability distribution, μ represents its mean vector, σ represents the standard deviation vector corresponding to the diagonal variance of its covariance matrix, and the present invention designs the following Figure 3 The expert product fusion module based on the view missing index information is shown to fuse the view features of multiple views:

[0068]

[0069]

[0070] in, is the fusion feature of the i-th input sample, M is the view missing index matrix, if the v-th view of the i-th input sample is not missing, then M iv =1, otherwise M iv =0.

[0071] Feature consistency maintenance module: There are two main challenges to deal with when fusing features from multiple perspectives. On the one hand, when performing batch training, the difference in the distribution of missing views between samples in different batches may cause the view-specific probability encoders corresponding to different perspectives to be unbalancedly trained. On the other hand, the inherent information capacity difference between perspectives of multi-view data will aggravate the above-mentioned imbalanced training situation. Specifically, unbalanced training will cause the fusion features to be overly dependent on some view features and ignore other view features. During the model training process, the neglected view features contribute very little to the loss calculation. When using the stochastic gradient descent algorithm to update the model parameters, the view-specific probability encoders corresponding to these view points cannot obtain enough gradients for parameter updates, making it impossible to fully extract the information of the corresponding view, and ultimately making it impossible for the fusion features to fully mine and utilize the information of each view, resulting in suboptimal clustering results. In order to solve the above challenges, such as Figure 4 As shown, the present invention designs a feature consistency maintenance module, in which the following feature consistency loss function based on the relative entropy between the corresponding distribution of fusion features and the corresponding distribution of view features is adopted:

[0072]

[0073] Among them, L ch Represents the feature consistency loss function, N represents the total number of samples of incomplete multi-view data, V is the total number of perspectives of the input data, The perspective feature obtained by encoding the v-th perspective data of the i-th incomplete multi-view input sample, is the fusion feature of the i-th input sample, M is the view missing index matrix, if the v-th view of the i-th input sample is missing, then M iv =1, otherwise M iv =0.

[0074] like Figure 5As shown, the feature consistency maintenance module promotes the fusion feature to fully contain the consistency information between different perspectives by minimizing the feature consistency loss between the fusion feature and each perspective feature. Cluster allocation module: The cluster allocation module adopts sampling approximation technology. After obtaining the fusion feature through the above-mentioned multi-perspective feature fusion module, it uses the resampling technology to sample a feature sample from the corresponding distribution of the fusion feature, calculate the probability of the sample under the prior feature distribution of different categories, and obtain the approximate clustering probability by performing normalization operation on the probabilities under different categories; in addition, in order to promote the model to learn fusion features with better clustering structure, the present invention designs a feature clustering structure loss function. The feature clustering structure loss function uses the above-mentioned approximate clustering probability as a weight, and calculates the weighted relative entropy between the corresponding distribution of the fusion feature and the prior feature distribution of each category, which is specifically expressed as:

[0075]

[0076] in,

[0077]

[0078] γ ic Represents the feature sample z sampled from the corresponding distribution of the i-th fusion feature i The approximate probability of belonging to the cth category, ε i represents a sample randomly sampled from a multivariate Gaussian distribution with mean 0 and covariance matrix equal to the identity matrix, μ c represents the mean of the prior feature distribution of the cth category, σ c Represents the standard deviation of the prior feature distribution of the cth category, Represents the Hadamard product, there are C {μ c ; σ c} corresponds to C categories of prior feature distribution, As a learnable parameter, it participates in the parameter update during model training. After the model training is completed, the cluster assignment of the i-th incomplete multi-view input sample can be obtained by the following formula:

[0079]

[0080] Deep multi-view probabilistic decoding network: The deep multi-view probabilistic decoding network designed by this invention mainly consists of V independent view-specific probabilistic decoders, each of which uses samples sampled from the corresponding distribution of fused features. Decoding gets the reconstructed data of the corresponding perspective, Represents a sample randomly sampled from a multivariate Gaussian distribution with a mean of 0 and a covariance matrix of the identity matrix, which promotes the fusion feature encoding of multi-view by minimizing the reconstruction loss between the original data and the reconstructed data at each viewpoint. Figure 1 Specifically, in order to obtain the reconstructed data generated by the feature samples at each viewing angle, the present invention designs the probability decoder of each viewing angle to be the opposite structure of the corresponding viewing angle probability encoder, and its structure is as follows: Figure 6 As shown in Figure 2, each probability decoder consists of four linear layers and three linear rectifier activation functions (ReLU). For the probability decoder of the vth view, the output dimensions of its linear layer are set to 2000, 500, 500, and d respectively. v , where d v is the original data dimension of the vth perspective. For different data value types, the present invention designs different incomplete multi-view reconstruction loss functions. Taking the vth perspective as an example, if its data value is a real number, the incomplete multi-view data reconstruction loss function can be expressed as:

[0081]

[0082] in, Represents the original data of the vth perspective of the i-th input sample, Represents the reconstructed data generated by decoding the original data through the model;

[0083] If the input original data value is binary type, the loss function for incomplete multi-view data reconstruction is calculated as:

[0084]

[0085] Here, Sigmoid represents the Sigmoid function.

[0086] The complete model training loss function of the present invention: The model training loss function of the present invention consists of three parts, namely the incomplete multi-view data reconstruction loss function Feature clustering structure loss function L KL And feature consistency loss function L ch , so the model training loss function of the incomplete multi-view clustering system of the present invention is expressed as:

[0087]

[0088] In the above model training loss function, α and β are adjustable hyperparameters. By jointly optimizing the model training loss function L, the optimal deep incomplete multi-view clustering model of the present invention can be obtained.

[0089] The present invention also discloses an incomplete multi-view clustering method based on a variational autoencoder, which includes two stages: a model parameter initialization stage and a model training loss function optimization stage.

[0090] In order to facilitate the subsequent explanation, we first give the definition of the symbols involved: the original multi-view data is represented by It represents that there are V perspectives and N samples, and the binary matrix M∈{0,1} N×V Records the missing information of the incomplete multi-view data, M iv = 0 means that the vth view data of the i-th incomplete multi-view sample is missing; otherwise, M iv =1, indicating that the v-th perspective data is not missing.

[0091] The model parameter initialization phase includes the following steps:

[0092] The first step is initialization: first, fill the missing view data in the original incomplete multi-view data with 0 values, record all the single-view samples that are not missing under each view, the complete multi-view samples without missing view data, and the incomplete multi-view samples with missing view data, set the data iterator, load 256 samples per iteration, and set the target number of clusters C;

[0093] The second step is to initialize the parameters of the encoding-decoding network: for each viewpoint, the probability encoder-probabilistic decoder uses all the non-missing single-view samples under the corresponding viewpoint, encodes the features through the probability encoder, and decodes the features with the probability decoder to generate reconstructed data, and calculates the mean square error loss between the original data and the reconstructed data. in represents the number of non-missing single-view samples at the v-th view. The training goal is to minimize the mean square error loss between the reconstructed data and the original data at each view. The parameters of the probability encoder-probability decoder are iteratively updated. The number of training iterations is set to 200 rounds. j represents the sample subscript;

[0094] The third step is to initialize the category prior feature distribution: After completing the encoder-decoder parameter initialization in the second step, the complete multi-view sample without missing view data is fed into the model, and the data features under each view are encoded by the view-specific probability encoder. The features of different views of the same sample are added and averaged to obtain Average features, N com Represents the number of complete multi-view samples without missing view data, for N com The K-means algorithm is executed on the average features, and the C cluster centers obtained after the execution are set as the mean of the category prior feature distribution and All are set to 1 to complete the model parameter initialization;

[0095] The model training loss function optimization stage includes:

[0096] Step 1: Set hyperparameters: Set the learning rate of the training process to 0.0005, the number of training rounds to 300, and the hyperparameters α = 1 and β = 10;

[0097] Step 2: Multi-view feature fusion: Based on the deep multi-view probability coding network module and the multi-view feature fusion module, the incomplete multi-view data is encoded by each view-specific probability encoder to obtain the corresponding view features. Then, according to formulas (1) and (2), the fusion feature is obtained by the expert product method.

[0098] Step 3: Multi-view data reconstruction: Using reparameterization techniques, sample from the corresponding distribution of fusion features The feature samples obtained Input the deep multi-view probability decoding network module to decode and generate the reconstructed data of each view

[0099] Step 4: Generate pseudo labels for feature categories: Use reparameterization techniques to sample from the corresponding distribution of fusion features Using the obtained feature sample z i , calculate the approximate probability of belonging to each category according to formula (5)

[0100] Step 5: Model training loss calculation: According to L KL and L ch , using formula (9), calculate the model training loss

[0101] Step 6: Gradient feedback optimization: Based on the model training loss obtained in step 5, use the stochastic gradient descent optimization algorithm to update all learnable parameters in the model.

[0102] Step 7, iterative training: according to the training rounds set in step 1, repeat the model training process from step 2 to step 6. After the iteration is completed, the final incomplete multi-view clustering model is obtained;

[0103] Experimental results on four incomplete multi-view datasets compared with the most advanced completion-based and non-completion-free algorithms show that the proposed method can achieve the best clustering performance.

[0104] Table 1 shows the average clustering accuracy (ACC) obtained on four datasets when the view missing rate is 50%. DCP is an incomplete multi-view clustering method based on completion, while DIMVC is an incomplete multi-view clustering method without completion.

[0105] Table 1

[0106] Dataset DCP DIMVC The present invention Caltech7-5V 0.452 0.761 0.868 Scene-15 0.383 0.413 0.443 Multi-Fashion 0.608 0.692 0.824 NoisyMNIST 0.894 0.633 0.923

[0107] The present invention also discloses a computer-readable storage medium storing a computer program configured to implement the steps of the method of the present invention when called by a processor.

[0108] The beneficial effects of the present invention are: 1) The present invention applies the probabilistic encoder to the incomplete multi-view clustering task for the first time, and can flexibly and efficiently handle clustering tasks with arbitrary missing view data without the need for additional data completion steps. The complexity of the proposed model is linearly related to the number of viewpoints and can be applied to scenes with multiple viewpoints; 2) The present invention adopts a distribution-based clustering prediction method. The clustering prediction results can be directly obtained in the proposed model without the need to perform additional clustering algorithms on the extracted features to obtain the final results. Therefore, the present invention is more likely to obtain better clustering performance; 3) The experimental results of comparing the present invention with the most advanced completion-based and non-completion-free algorithms on four incomplete multi-view datasets show that the present invention can obtain the best clustering performance.

[0109] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. An incomplete multi-view clustering system based on variational autoencoder, characterized by: It includes a deep multi-view probability coding network module, a multi-view feature fusion module, a feature consistency maintenance module, a clustering allocation module and a deep multi-view probability decoding network module, wherein: The deep multi-view probabilistic coding network module maps the original high-dimensional data encoding to a low-dimensional feature space and obtains the feature distribution of the data, thereby extracting condensed information from the original multi-view data with redundant information; The multi-view feature fusion module treats data features from different viewpoints as viewpoint experts and uses an expert product method to effectively fuse data features from all viewpoints when any viewpoint data is missing, thereby obtaining fused features that are unanimously recognized by experts from different viewpoints and contain shared information between multiple viewpoints. The feature consistency maintenance module reduces the distribution difference between the fused features and the features of each view, alleviating the information imbalance problem that may exist in the fused features when some view data is missing. The cluster assignment module infers cluster assignments by fused features, generates category pseudo-labels during model training to guide the model to learn features with a good cluster structure, and directly obtains clustering results through the cluster assignment module after model training is completed. The deep multi-view probabilistic decoding network module: the decoding network generates multi-view reconstructed data based on the fusion feature decoding, and minimizes the reconstruction loss between the original data and the reconstructed data so that the fusion feature retains more information; The data processed by the incomplete multi-view clustering system includes image data, audio data, and text data; in the multi-view feature fusion module, the output obtained by the deep multi-view probability coding network module of the incomplete multi-view data is defined as Where N is the total number of samples of incomplete multi-view data, V represents the total number of viewpoints of the input data, The view feature obtained by encoding the vth view data of the i-th incomplete multi-view input sample corresponds to a probability distribution, μ represents its mean vector, σ represents the standard deviation vector corresponding to the diagonal variance of its covariance matrix, and the multi-view feature fusion module based on the view missing index information is used to fuse the view features of multiple views to obtain: in, is the fusion feature of the i-th input sample, M is the perspective missing index matrix, if the v-th perspective data of the i-th input sample is missing, then M iv =0, otherwise M iv =1; In the feature consistency maintenance module, the following feature consistency loss function based on the relative entropy between the corresponding distribution of the fusion feature and the corresponding distribution of the view feature is adopted:

2. The incomplete multi-view clustering system according to claim 1, characterized in that: The deep multi-view probabilistic coding network module consists of a view-specific probabilistic encoder corresponding to each view, and the probabilistic encoder structure under different views can be expanded according to the data input type of the corresponding view.

3. The incomplete multi-view clustering system according to claim 2, characterized in that: The view-specific probabilistic encoder is constructed based on fully connected linear layers. Each probabilistic encoder consists of four layers. Each of the first three layers consists of a linear layer, and the fourth layer consists of two independent linear layers with the same input and output dimensions. The two linear layers in the fourth layer encode the mean and standard deviation of the feature distribution respectively. The linear layer encoding the standard deviation is followed by a Softplus activation function to match the non-negative property of the standard deviation. In each probabilistic encoder, two adjacent layers are connected in series by a linear rectification activation function. The output dimensions of the linear layer are set to 500, 500, 2000, and d, respectively, where d is the feature dimension of the desired data feature.

4. The incomplete multi-view clustering system according to claim 3, characterized in that: After obtaining the fused features through the multi-view feature fusion module, the cluster assignment module uses a resampling technique to sample a feature sample from the corresponding distribution of the fused features, calculates the probability of the sample under the prior feature distribution of different categories, and obtains an approximate clustering probability by performing a normalization operation on the probabilities under all categories; In the cluster assignment module, in order to promote the model to learn fusion features with good clustering structure, a feature clustering structure loss function is designed. The feature clustering structure loss function uses the approximate clustering probability as the weight and calculates the weighted relative entropy between the corresponding distribution of the fusion feature and the prior feature distribution of each category. It is specifically expressed as: in, γ ic Represents the feature sample z sampled from the corresponding distribution of the i-th fusion feature i The approximate probability of belonging to the cth category, ε i represents a sample randomly sampled from a multivariate Gaussian distribution with mean 0 and covariance matrix equal to the identity matrix, μ c represents the mean of the prior feature distribution of the cth category, σ c Represents the standard deviation of the prior feature distribution of the cth category, there are C {μ c ; σ c } corresponds to C categories of prior feature distribution, As a learnable parameter, it participates in the parameter update during model training. After the model training is completed, the cluster assignment of the i-th incomplete multi-view input sample is obtained by the following formula:

5. The incomplete multi-view clustering system according to claim 4, characterized in that: The deep multi-view probabilistic decoding network module consists of multiple independent view-specific probabilistic decoders, each of which uses feature samples sampled from the corresponding distribution of fused features. Decoding generates reconstructed data corresponding to the viewing angle, It represents a sample randomly sampled from a multivariate Gaussian distribution with a mean of 0 and a covariance matrix of the identity matrix. It promotes the fusion feature encoding of multi-view consistent and clustering-friendly information by minimizing the reconstruction loss between the original data and the reconstructed data at each perspective.

6. The incomplete multi-view clustering system according to claim 5, characterized in that: In the deep multi-view probabilistic decoding network module, the view-specific probabilistic decoder of each view is designed to have a structure opposite to that of the corresponding view probabilistic encoder. Each probabilistic decoder contains four linear layers, and two adjacent linear layers are connected in series by a linear rectification activation function. For the view-specific probabilistic decoder of the vth view, the output dimensions of each linear layer are set to 2000, 500, 500 and d respectively. v , where d v is the original data dimension of the v-th perspective; In the deep multi-view probabilistic decoding network module, different incomplete multi-view data reconstruction loss functions are designed for different types of original data values. In the vth view, if the input original data value is a real number, the incomplete multi-view data reconstruction loss function is expressed as: in, Represents the original data of the vth perspective of the i-th input sample, Represents the reconstructed data generated by decoding the original data through the model; if the input original data value is binary type, the incomplete multi-view data reconstruction loss function is expressed as: The model training loss function of the incomplete multi-view clustering system is expressed as: Among them, α and β are adjustable hyperparameters.

7. An incomplete multi-view clustering method based on variational autoencoder, characterized in that The incomplete multi-view clustering system according to claim 6 includes a model parameter initialization phase and a model training loss function optimization phase, wherein the model parameter initialization phase includes the following steps: The first step is initialization: first, fill the missing view data in the original incomplete multi-view data with 0 values, record all the single-view samples that are not missing under each view, the complete multi-view samples without missing view data, and the incomplete multi-view samples with missing view data, set the data iterator, load the required number of sample data in each iteration, and set the target number of clusters C; The second step is to initialize the parameters of the encoding-decoding network: for each viewpoint, the probability encoder-probabilistic decoder uses all the non-missing single-view samples under the corresponding viewpoint, encodes the features through the probability encoder, and decodes the features with the probability decoder to generate reconstructed data, and calculates the mean square error loss between the original data and the reconstructed data. in represents the number of non-missing single-view samples at the v-th view. The training goal is to minimize the mean square error loss between the reconstructed data and the original data at each view. The parameters of the probability encoder-probability decoder are iteratively updated, and the number of training iterations is set as required. j represents the sample subscript; The third step is to initialize the category prior feature distribution: a complete multi-view sample without missing view data is fed into the model, the data features of each view are encoded through the view-specific probability encoder, and the features of different viewpoints of the same sample are added and averaged to obtain Average features, N com Represents the number of complete multi-view samples without missing view data, for N com The K-means algorithm is executed on the average features, and the C cluster centers obtained after the execution are set as the mean of the category prior feature distribution and All are set to 1 to complete the model parameter initialization; The model training loss function optimization stage includes: Step 1: Set hyperparameters: Set the learning rate of the training process to the desired value, set the desired number of training rounds, and hyperparameters α = 1 and β = 10; Step 2: Multi-view feature fusion: Based on the deep multi-view probability coding network module and the multi-view feature fusion module, the incomplete multi-view data is encoded by each view-specific probability encoder to obtain the corresponding view features. Then, according to formulas (1) and (2), the fusion feature is obtained by the expert product method. Step 3: Multi-view data reconstruction: Using reparameterization techniques, sample from the corresponding distribution of its fusion features The feature samples obtained Input the deep multi-view probability decoding network module to decode and generate the reconstructed data of each view Step 4: Generate pseudo labels for feature categories: Use reparameterization techniques to sample from the corresponding distribution of fusion features Using the obtained feature sample z i , calculate the approximate probability of belonging to each category according to formula (5) Step 5: Model training loss calculation: According to L ch , L KL 、 Using formula (9), calculate the model training loss Step 6: Gradient feedback optimization: Based on the model training loss obtained in step 5, use the stochastic gradient descent optimization algorithm to update all learnable parameters in the model. Step 7, iterative training; according to the training rounds set in step 1, repeat the model training process from step 2 to step 6. After the iteration is completed, the final incomplete multi-view clustering model is obtained; the data processed by this incomplete multi-view clustering method includes image data, audio data, and text data.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the method according to claim 7 when called by a processor.

Images