Missing multi-view face image clustering method based on double alignment completion

Abstract

A missing multi-view face image clustering method based on double alignment completion belongs to the field of data clustering processing in missing multi-view face image data, and comprises the following steps: firstly, obtaining low-dimensional embedded representation of a missing multi-view face image data sample through an encoding network of an auto-encoder; and obtaining reconstructed sample features through a decoding network, and constraining the missing multi-view face image data and the reconstructed sample features by using a reconstruction loss function. Then, a prototype matrix is constructed by utilizing the potential embedding representation of each view, and similar cluster prototypes between the views are aligned through a prototype comparison loss function; and inputting the potential embedded representation of each view and the corresponding similar cluster prototype into an attention layer, increasing the tightness of similar cluster samples in the views through an attention matrix, and complementing missing views by using similar cluster existing samples among the views. Next, the complemented views are subjected to instance contrast learning to obtain consistency information of multi-view face image data samples among the views, and then common low-dimensional embedding representations of all the views are obtained through a feature fusion layer. And finally, using a small-batch stochastic gradient descent algorithm to optimize the total loss function of the missing multi-view face image clustering method based on double-alignment completion to convergence. And after the model achieves convergence, clustering the common low-dimensional embedded representation by using a k-mean algorithm to obtain a clustering result, thereby achieving the purpose of clustering missing multi-view face image data samples.

Description

Technical Field

[0001] The present invention belongs to the field of data clustering processing in missing multi-view face image data, and in particular relates to a missing multi-view face image clustering method based on double alignment completion. Background Art

[0002] In the field of multimedia information processing, face image data is not only a widely available data, but also can be collected from multiple different angles or channels to describe the same sample. Such face image data with multiple modes or collected from multiple different sources is called multi-view face image data. The key to clustering such data is how to mine the correlation between different views and the knowledge in each view from multi-view data, so as to discover the rules of multi-view data and better cluster them. However, since multi-view face image data may have information missing or some channels may fail in various stages such as collection, transmission, and storage, the integrity of multi-view face image data is destroyed. Therefore, the study of clustering of missing multi-view face image data has gradually become an indispensable part of multi-view learning. In addition, the missing view not only seriously affects the accuracy of face image data in the clustering process, but also destroys the consistency of the spatial dimension of multi-view face image data, increasing the difficulty of data fusion. At present, the most common method for processing missing face image data is mean or zero filling, that is, filling the missing sample data according to the mean of the existing data sample or using zero value. However, this method may introduce additional error information or outliers to the original data, thereby affecting the clustering accuracy of the model. Based on this, the present invention proposes a missing multi-view face image clustering method based on double alignment completion, which aligns the prototypes and samples in the same cluster from the two aspects of prototype and instance respectively, and completes the missing view by introducing an attention layer, so that the completed missing view samples are closer to the real samples, thereby achieving the purpose of accurately clustering the missing multi-view face image data. Summary of the invention

[0003] In order to solve the above problems, the present invention provides a missing multi-view face image clustering method based on double alignment completion, the method comprising the steps of:

[0004] First, the multi-view face image data in each view is normalized and then input into the encoding network to obtain the latent low-dimensional embedding representation of each view. Then, the latent low-dimensional embedding representation is passed through the decoding network to obtain the reconstructed sample features. Finally, the reconstruction loss function is used to constrain the multi-view face image data and the reconstructed sample features.

[0005] The existing samples in a view are used to construct a prototype matrix for the view in a low-dimensional embedding space, and the prototypes of the same cluster between views are aligned using a cross-view prototype contrast loss function.

[0006] The low-dimensional embedding representation of each view and the corresponding prototype matrix are input into the attention layer. In the attention layer, the attention matrix is ​​used to increase the compactness of samples of the same cluster within the view on the one hand, and to complete the missing views through the existing samples of the same cluster between views on the other hand.

[0007] The completed view samples are aligned through the instance contrast loss function to mine the consistency information of samples between views, and the completed view samples are fused through the feature fusion layer to obtain a common low-dimensional embedding representation for all views.

[0008] The mini-batch stochastic gradient descent algorithm is used to optimize the overall loss function of the missing multi-view face image clustering method based on double alignment completion to convergence, and then the final clustering result is obtained.

[0009] Based on the obtained clustering results, the clustering accuracy on the missing multi-view face image data is calculated.

[0010] Furthermore, the method normalizes the multi-view face image data samples in each view and inputs them into the encoding network to obtain a low-dimensional embedding representation of each view, and passes it through the decoding network to obtain reconstructed sample features, while using a reconstruction loss function to constrain the multi-view face image data and the reconstructed sample features.

[0011] For each multi-view face image data sample in each view, its feature vector is normalized to a standard format with a mean of 0 and a standard deviation of 1. The proposed normalized expression is as follows:

[0012]

[0013] Among them, i represents the i-th existing sample, v represents the v-th view, represents the feature vector of the i-th sample in the v-th view, and Respectively The mean and standard deviation of .

[0014] The normalized multi-view face image data is input into the encoding network and the decoding network in sequence to obtain the reconstructed sample features. The expression of the proposed encoding network is:

[0015]

[0016] in, represents the encoding network of the vth view, X vrepresents the face image data sample of the vth view, θ v N represents the learnable parameters of the encoding network for the vth view. v represents the number of samples in the vth view, D v represents the feature dimension of the v-th view, Z v represents the low-dimensional embedding representation of the v-th view, and D represents the feature dimension of the latent embedding representation.

[0017] The expression of the proposed decoding network is:

[0018]

[0019] in, represents the decoding network of the vth view, φ v represents the learnable parameters of the decoding network of the vth view, Represents the reconstructed sample features of the vth view.

[0020] The reconstruction loss function is used to constrain the multi-view face image data samples and the reconstructed sample features. The expression of the proposed reconstruction loss function is:

[0021]

[0022] in, represents the reconstruction loss function, V represents the number of views, X v represents the face image data sample of the vth view, Z v represents the low-dimensional embedding representation of the v-th view, represents the decoding network of the vth view, ‖·‖2 represents the L2 norm of the matrix, N v represents the number of samples in the vth view, represents the feature vector of the i-th sample in the v-th view, represents the encoding network of the vth view.

[0023] Furthermore, in the low-dimensional embedding space, a prototype matrix is ​​constructed for each view using existing samples in the view, and prototypes of the same cluster between views are aligned through a prototype contrast loss function across views.

[0024] Construct a prototype matrix for each view in the low-dimensional embedding space, and the proposed expression formula is:

[0025]

[0026] Among them, P v represents the prototype matrix of the v-th view, represents the jth prototype in the vth view, represents the low-dimensional embedding representation of the i-th sample in the v-th view, K represents the number of prototypes, N v Represents the number of samples present in the vth view.

[0027] The prototypes of the same cluster between views are aligned through the prototype contrast loss function across views. The proposed expression formula is:

[0028]

[0029] in, represents the prototype contrast loss function for all views, represents the contrast loss of the c-th prototype in the v-th view, represents the cth prototype in the vth view, express The positive sample corresponding to the prototype in the rth view, express The prototype is the negative sample corresponding to the rth view, S(·) represents the cosine similarity function, τ represents the temperature coefficient, V represents the number of views, and K represents the number of prototypes.

[0030] Furthermore, the low-dimensional embedding representation of each view and the corresponding prototype matrix are input into the attention layer. The attention matrix is ​​used to increase the compactness of samples of the same cluster within the view on the one hand, and to complete the missing views through the existing samples of the same cluster between views on the other hand.

[0031] The low-dimensional embedding representation of each view is input into the attention layer to increase the compactness of similar cluster samples in the view. The proposed expression is:

[0032]

[0033] in, Represents the linear layer of the prototype matrix corresponding to the v-th view, Z v A represents the low-dimensional embedding representation of the v-th view. v represents the attention matrix of the v-th view, and its expression formula is:

[0034]

[0035] in, Represents the low-dimensional embedding representation Z in the v-th view v The linear layer, Represents the prototype matrix P in the vth view v The linear layer of , D represents the feature dimension of the potential embedding representation, and Softmax( ) represents the Softmax activation function.

[0036] For the attention matrix A vRegularization constraint is performed, and its expression formula is:

[0037]

[0038] in, represents the probability that the sample in the vth view belongs to the jth category, represents the probability that the i-th sample in the v-th view belongs to the j-th category, and β is the balance parameter.

[0039] In the low-dimensional embedding representation, the existing samples of the same cluster between views are used to complete the missing views. The proposed expression is:

[0040]

[0041] Among them, H v represents the complete low-dimensional embedding representation of the v-th view, Represents the linear layer of the prototype matrix corresponding to the v-th view, Z m A represents the low-dimensional embedding representation of the sample in the m-th view. m represents the attention matrix of the m-th view, P v Represents the prototype matrix of the v-th view.

[0042] Furthermore, the completed view samples are aligned through an instance contrast loss function to mine the consistency information of samples between views, and the completed view samples are fused through a feature fusion layer to obtain a common low-dimensional embedding representation of all views.

[0043] The completed view is used to mine the consistency information of samples between views through the instance contrast loss function. The proposed expression is:

[0044]

[0045] in, represents the instance contrast loss function for all views, represents the contrast loss of the i-th instance in the v-th view, represents the i-th instance in the v-th view, express The positive sample corresponding to the instance in the rth view, express The negative sample corresponding to the instance in the t-th view, S(·) represents the cosine similarity function, τ represents the temperature coefficient, V represents the number of views, and N represents the number of instances.

[0046] The complete low-dimensional embedding representation is passed through the feature fusion layer to obtain the common low-dimensional embedding representation of all views. The proposed expression is:

[0047]

[0048] sth ij ≥0,1 T h i =1

[0049] Among them, h ij ∈H represents the common low-dimensional embedding representation of all views, H v represents the low-dimensional embedding representation of the vth complete view, and V represents the number of views.

[0050] Furthermore, the mini-batch stochastic gradient descent algorithm is used to optimize the overall loss function of the missing multi-view face image clustering method based on dual alignment completion to convergence. The reconstruction loss function, the attention matrix regularization constraint, the prototype contrast loss function and the instance contrast loss function are summed to obtain the overall loss function of the missing multi-view face image clustering method, which is expressed as:

[0051]

[0052] in, represents the reconstruction loss function, represents the attention matrix regularization constraint, represents the prototype contrast loss function, represents the instance contrastive loss function.

[0053] Furthermore, after the model reaches convergence, the k-mean algorithm is used to cluster the common low-dimensional embedding representation to obtain a clustering result, and the clustering accuracy on the missing multi-view face image data is calculated based on the obtained clustering result.

[0054] The present invention provides a missing multi-view face image clustering method based on double alignment completion, which has the following advantages:

[0055] (1) The method adopts a missing multi-view clustering framework, which makes full use of the consistency and complementary information between views in missing multi-view face image data, and can effectively mine the potential cluster structure of the data, thereby achieving accurate clustering.

[0056] (2) The method adopts a technology that combines an autoencoder network and dual alignment learning. The autoencoder network is used to construct a low-dimensional embedding representation of the view. Then, prototype comparison and instance comparison are used on this basis to align the consistency information of prototypes and samples of the same cluster, thereby effectively reducing the impact of missing views on the clustering results.

[0057] (3) The method uses an attention layer to complete the missing multi-view facial image data. By constructing an attention matrix in the attention layer, on the one hand, the compactness of the same cluster samples within the view is increased, and on the other hand, the same cluster samples in different views are used to complete the missing data. By introducing the attention layer, the completed missing multi-view facial image data can be made closer to the real sample data, thereby achieving the purpose of accurately clustering the missing multi-view facial image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the drawings without paying creative labor.

[0059] Figure 1 It is a flow chart of the missing multi-view face image clustering method based on double alignment completion provided by the present invention.

[0060] Figure 2 This is a schematic diagram of part of the ORL face image dataset. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0062] Exemplary Methods

[0063] like Figure 1 As shown, after the processing flow starts, step S110 is first executed.

[0064] Step S110: normalize the data samples in the view and input them into the encoding network and the decoding network in sequence to obtain reconstructed sample features.

[0065] For each multi-view face image data sample in each view, its feature vector is normalized. As an example, the normalization defined in step S110 is as follows:

[0066]

[0067] Among them, i represents the i-th existing sample, v represents the v-th view, represents the feature vector of the i-th sample in the v-th view, and Respectively The mean and standard deviation of .

[0068] The normalized multi-view face image data samples are sequentially input into the encoding network and the decoding network to obtain the reconstructed sample features. As an example, the expression of the encoding network defined in step S110 is as follows:

[0069]

[0070] in, represents the encoding network of the vth view, X v represents the face image data sample of the vth view, θ v N represents the learnable parameters of the encoding network for the vth view. v represents the number of samples in the vth view, D v represents the feature dimension of the v-th view, Z v represents the low-dimensional embedding representation of the v-th view, and D represents the feature dimension of the latent embedding representation.

[0071] As an example, the expression of the decoding network defined in step S110 is as follows:

[0072]

[0073] in, represents the decoding network of the vth view, φ v represents the learnable parameters of the decoding network of the vth view, Represents the reconstructed sample features of the vth view.

[0074] The reconstruction loss function is used to constrain the multi-view face image data samples and the reconstructed sample features. As an example, the reconstruction loss function expression is defined as:

[0075]

[0076] in, represents the reconstruction loss function, V represents the number of views, X v represents the face image data sample of the vth view, Z v represents the low-dimensional embedding representation of the v-th view, represents the decoding network of the vth view, ‖·‖2 represents the L2 norm of the matrix, N v represents the number of samples in the vth view, represents the feature vector of the i-th sample in the v-th view, represents the encoding network of the vth view.

[0077] Step S120: construct a prototype matrix for each view, and align prototypes of similar clusters between views using a prototype contrast loss function.

[0078] In the low-dimensional embedding space, the existing samples in each view are used to construct a prototype matrix for the view. As an example, the view prototype matrix construction expression defined in step S120 is as follows:

[0079]

[0080] Among them, P v represents the prototype matrix of the v-th view, represents the jth prototype in the vth view, represents the low-dimensional embedding representation of the i-th sample in the v-th view, K represents the number of prototypes, N v Represents the number of samples present in the vth view.

[0081] Prototypes of the same cluster between views are aligned by using a prototype contrast loss function across views. As an example, the prototype contrast loss function expression defined in step S120 is as follows:

[0082]

[0083] in, represents the prototype contrast loss function for all views, represents the contrast loss of the c-th prototype in the v-th view, represents the cth prototype in the vth view, express The positive sample corresponding to the prototype in the rth view, express The prototype is the negative sample corresponding to the rth view, S(·) represents the cosine similarity function, V represents the number of views, K represents the number of prototypes, and τ represents the temperature coefficient, which is 1 in the present invention.

[0084] Step S130: Input the low-dimensional embedding representation into the attention layer to increase the compactness of similar cluster samples and complete the missing views.

[0085] The low-dimensional embedding representation of each view is input into the attention layer to increase the compactness of similar cluster samples in the view. As an example, the sample compactness expression defined in step S130 is as follows:

[0086]

[0087] in, Represents the linear layer of the prototype matrix corresponding to the v-th view, Z v A represents the low-dimensional embedding representation of the v-th view. vrepresents the attention matrix of the v-th view, and its expression formula is:

[0088]

[0089] Among them, linear i v (·) represents the low-dimensional embedding representation Z in the v-th view v The linear layer, Represents the prototype matrix P in the vth view v The linear layer of , D represents the feature dimension of the potential embedding representation, and Softmax( ) represents the Softmax activation function.

[0090] For the attention matrix A v Regularization constraint is performed, and its expression formula is:

[0091]

[0092] in, represents the probability that the sample in the vth view belongs to the jth category, represents the probability that the i-th sample in the v-th view belongs to the j-th category, and β is a balancing parameter, which is 0.1 in the present invention.

[0093] In the low-dimensional embedding representation, the missing views are completed using the existing samples of the same cluster between views. As an example, the expression for completing the missing views defined in step S130 is as follows:

[0094]

[0095] Among them, H v represents the complete low-dimensional embedding representation of the v-th view, represents the linear layer of the prototype corresponding to the vth missing view, Z m A represents the low-dimensional embedding representation of the sample in the m-th view. m represents the attention matrix of the m-th view, P v Represents the prototype matrix of the v-th view.

[0096] Step S140: Obtain consistency information of the complete view through the instance contrast loss function, and obtain a common low-dimensional embedding representation through the feature fusion layer.

[0097] The completed view is used to mine the consistency information of samples between views through the instance contrast loss function. As an example, the expression of the instance contrast loss function defined in step S140 is as follows:

[0098]

[0099] in, represents the instance contrast loss function for all views, represents the contrast loss of the i-th instance in the v-th view, represents the i-th instance in the v-th view, express The positive sample corresponding to the instance in the rth view, express The negative sample corresponding to the instance in the t-th view, S(·) represents the cosine similarity function, V represents the number of views, N represents the number of instances, and τ represents the temperature coefficient, which is 1 in the present invention.

[0100] The complete low-dimensional embedding representation is passed through a feature fusion layer to obtain a common low-dimensional embedding representation for all views. As an example, the feature fusion layer expression defined in step S140 is as follows:

[0101]

[0102] Among them, h ij ∈H represents the common low-dimensional embedding representation of all views, H v Represents the low-dimensional embedding representation of the v-th complete view, where V is the number of views.

[0103] Step S150: using a mini-batch stochastic gradient descent algorithm to optimize the overall loss function of the missing multi-view face image clustering method based on double alignment completion until convergence.

[0104] The reconstruction loss function, the attention matrix regularization constraint, the prototype contrast loss function and the instance contrast loss function are summed to obtain the overall loss function of the missing multi-view face image clustering method based on double alignment completion. As an example, the overall loss function expression of the missing multi-view face image clustering method based on double alignment completion defined in step S150 is as follows:

[0105]

[0106] in,

[0107] represents the reconstruction loss function, represents the attention matrix regularization constraint, represents the prototype contrast loss function, represents the instance contrastive loss function.

[0108] Step S160: Calculate the clustering accuracy on the missing multi-view face image data according to the obtained clustering results.

[0109] As an example, after the model reaches convergence in step S160, the k-mean algorithm is used to cluster the common low-dimensional embedding representation to obtain a clustering result, and the clustering accuracy on the missing multi-view face image data is calculated based on the obtained clustering result.

[0110] Through this implementation, firstly, the low-dimensional embedding representation of the missing multi-view face image data samples is obtained through the encoding network of the autoencoder, and the reconstructed sample features are obtained through the decoding network, and the reconstruction loss function is used to constrain the missing multi-view face image data and the reconstructed sample features. Then, the prototype matrix is ​​constructed using the potential embedding representation of each view, and the prototypes of the same clusters between views are aligned through the prototype contrast loss function. Then, the potential embedding representation of each view and the corresponding prototype of the same cluster are input into the attention layer, and the closeness of the samples of the same cluster in the view is increased through the attention matrix, and the missing views are supplemented with the existing samples of the same cluster between views. Then, the completed views are used to obtain the consistency information of the multi-view face image data samples between views through instance contrast learning, and then the common low-dimensional embedding representation of all views is obtained through the feature fusion layer. Finally, the overall loss function of the missing multi-view face image clustering method based on double alignment completion is optimized to convergence using a small batch stochastic gradient descent algorithm. When the model reaches convergence, the k-mean algorithm is used to cluster the common low-dimensional embedding representation to obtain the clustering result, thereby achieving the purpose of clustering the missing multi-view face image data samples.

[0111] Detailed description of the invention Results

[0112] This embodiment uses the publicly available ORL multi-view face image dataset and simulates missing multi-view face image data according to different sample missing ratios. The details of the ORL dataset are as follows:

[0113] The ORL face dataset contains 400 images from 40 different categories. The images are collected at different times, lighting, facial expressions (eyes open / closed, smiling / not smiling) and facial details (wearing glasses / not wearing glasses). All images are taken against a dark uniform background, with the subject standing upright and in a frontal position (some with a slight side tilt). The dataset has the following three views:

[0114] See Figure 1 The grayscale intensity view is the most basic image representation, and the value of each pixel represents the brightness of the pixel. In the ORL dataset, the size of each image is 92×112 pixels and the grayscale level is 256.

[0115] See Figure 2 It is a local binary pattern. A local binary pattern is a texture description operator that generates a binary pattern by comparing the grayscale values ​​of each pixel with its surrounding pixels.

[0116] View 3 is a Gabor filter, which is a linear filter that can capture edge, texture, and frequency information in an image.

[0117] In order to verify the superiority of the method of the present invention (Ours), the method of the present invention is compared with several common missing multi-view face image data clustering methods, including UEAF, SGC-IMC, UIMC, IMVC-CBG and other methods, and the clustering accuracy (ACC) of these methods on the above face image data is compared. The specific clustering results are shown in Table 1.

[0118] Table 1 ORL multi-view face image data clustering accuracy (%)

[0119]

[0120] From the comparison of clustering accuracy under different methods in the above table, it can be clearly seen that for multi-view face image data with different missing ratios, the method of the present invention (Ours) achieves the best performance index and has a relatively high clustering accuracy. The method of the present invention obtains the potential embedding representation of the sample by using an autoencoder, and uses double alignment learning in the latent space to align the prototype in each view and the samples between views, and completes the missing samples through the attention matrix, so that the completed samples are closer to the real samples, thereby achieving the purpose of accurately clustering the missing multi-view face image data.

[0121] This embodiment proposes a missing multi-view face image clustering method based on double alignment completion, which is used to perform cluster analysis on missing multi-view face data commonly seen in life. This method aligns the consistency information of prototypes and samples of the same cluster through double alignment learning, and completes the missing samples through the attention matrix, so that the completed samples are closer to the real samples, thereby achieving the purpose of accurately clustering the missing multi-view face image data. The experimental results on five missing sample ratios of missing multi-view face image data simulated using the ORL dataset show that this embodiment has a higher clustering accuracy and better superiority than other methods.

[0122] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims

1. A missing multi-view face image clustering method based on double alignment completion, characterized in that: The method comprises the steps of: S110. Normalize the facial image data samples in each view and input them into the encoding network to obtain a low-dimensional embedding representation of each view, and input them into the decoding network to obtain reconstructed sample features; S120. construct a prototype matrix for each view in the low-dimensional embedding space, and align similar cluster prototypes between views through a prototype contrast loss function; S130. Input the low-dimensional embedding representation of each view and the corresponding prototype matrix into the attention layer to increase the compactness of similar cluster samples in the view and complete the missing views; S140. The completed view is used to mine the consistency information of samples between views through the instance contrast loss function, and a common low-dimensional embedding representation of all views is obtained through the feature fusion layer; S150. Use a mini-batch stochastic gradient descent algorithm to optimize the overall loss function of the missing multi-view face image clustering method based on double alignment completion to convergence, thereby obtaining a clustering result; S160. Calculate the clustering accuracy on the missing multi-view face image dataset based on the obtained clustering results; In S110, for the existing multi-view face image data samples, the feature vectors are normalized, and the expression is as follows: Among them, i represents the i-th existing sample, v represents the v-th view, represents the feature vector of the i-th sample in the v-th view, and Respectively The mean and standard deviation of The normalized multi-view face image data samples are sequentially input into the encoding network and the decoding network to obtain the reconstructed sample features. The expression of the encoding network is as follows: in, represents the encoding network of the vth view, X v represents the face image data sample of the vth view, θ v N represents the learnable parameters of the encoding network for the vth view. v represents the number of samples in the vth view, D v represents the feature dimension of the v-th view, Z v represents the low-dimensional embedding representation of the v-th view, and D represents the feature dimension of the potential embedding representation; The expression of the decoding network is as follows: in, represents the decoding network of the vth view, φ v represents the learnable parameters of the decoding network of the vth view, Represents the reconstructed sample features of the vth view; The reconstruction loss function is used to constrain the multi-view face image data samples and the reconstructed sample features. The expression of the reconstruction loss function is as follows: in, represents the reconstruction loss function, V represents the number of views, X v represents the face image data sample of the vth view, Z v represents the low-dimensional embedding representation of the v-th view, represents the decoding network of the vth view, ‖·‖2 represents the L2 norm of the matrix, N v represents the number of samples in the vth view, represents the feature vector of the i-th sample in the v-th view, represents the encoding network of the vth view; In S120, a prototype matrix is ​​constructed for each view in the low-dimensional embedding space, and the expression formula is: Among them, P v represents the prototype matrix of the v-th view, represents the jth prototype in the vth view, represents the low-dimensional embedding representation of the i-th sample in the v-th view, K represents the number of prototypes, N v represents the number of samples in the vth view; The prototype contrast loss function is used to align the prototypes of similar clusters between views. The expression formula is: in, represents the prototype contrast loss function for all views, represents the contrast loss of the c-th prototype in the v-th view, represents the cth prototype in the vth view, express The positive sample corresponding to the prototype in the rth view, express The negative sample corresponding to the prototype in the rth view, S(·) represents the cosine similarity function, τ represents the temperature coefficient, V represents the number of views, and K represents the number of prototypes; In S130, the low-dimensional embedding representation of each view is input into the attention layer to increase the compactness of similar cluster samples in the view, and the expression formula is: in, Represents the linear layer of the prototype matrix corresponding to the v-th view, Z v A represents the low-dimensional embedding representation of the v-th view. v represents the attention matrix of the v-th view, and its expression formula is: in, Represents the low-dimensional embedding representation Z in the v-th view v The linear layer, Represents the prototype matrix P in the vth view v The linear layer of , D represents the feature dimension of the potential embedding representation, and Softmax( ) represents the Softmax activation function; For the attention matrix A v Regularization constraint is performed, and its expression formula is: in, represents the probability that the sample in the vth view belongs to the jth category, represents the probability that the i-th sample in the v-th view belongs to the j-th category, and β is the balance parameter; In the low-dimensional embedding representation, the missing views are completed using samples of the same cluster. The expression formula is: Among them, H v represents the low-dimensional embedding representation of the v-th complete view, Represents the linear layer of the prototype matrix corresponding to the v-th view, Z m A represents the low-dimensional embedding representation of the sample in the m-th view. m represents the attention matrix of the m-th view, P v Represents the prototype matrix of the v-th view; In S140, the completed view is used to mine the consistency information of the samples between views through the instance contrast loss function, which is expressed as: in, represents the instance contrast loss function for all views, represents the contrast loss of the i-th instance in the v-th view, represents the i-th instance in the v-th view, express The positive sample corresponding to the instance in the rth view, express The negative sample corresponding to the instance in the t-th view, S(·) represents the cosine similarity function, τ represents the temperature coefficient, V represents the number of views, and N represents the number of instances; The complete low-dimensional embedding representation is passed through the feature fusion layer to obtain the common low-dimensional embedding representation of all views. The expression formula is: s.t.h ij ≥0,1 T h i =1 Among them, h ij ∈H represents the common low-dimensional embedding representation of all views, H v represents the low-dimensional embedding representation of the vth complete view, where V represents the number of views; In S150, a mini-batch stochastic gradient descent algorithm is used to optimize the overall loss function of the missing multi-view face image clustering method based on dual alignment completion to convergence; the reconstruction loss function, the attention matrix regularization constraint, the prototype contrast loss function and the instance contrast loss function are summed to obtain the overall loss function of the missing multi-view face image clustering method based on dual alignment completion, and the expression formula is: in, represents the reconstruction loss function, represents the attention matrix regularization constraint, represents the prototype contrast loss function, represents the instance contrast loss function; In S160, when the model reaches convergence, the k-mean algorithm is used to cluster the common low-dimensional embedding representation to obtain a clustering result, and the clustering accuracy on the missing multi-view face image data is calculated based on the obtained clustering result.

Images