Multi-view data category identification method based on comparative learning and private information

Abstract

The invention discloses a multi-view data category identification method based on comparative learning and private information. The method comprises the following steps: obtaining mixed features from a plurality of input views by using an auto-encoder; obtaining a private feature set and a public semantic feature set according to the mixed features, performing same-view instance neighbor private feature learning by taking the private feature set as an object to obtain useful private features, and performing cross-view semantic information consistency feature learning by taking the public semantic feature set to obtain aligned public semantic features; and a K-Means clustering prediction model is introduced to generate useful pseudo labels of private features, corresponding constraint is carried out according to the pseudo labels of the private features and the aligned prediction labels of the public semantic features, and a final label with the maximum prediction probability average value in all views is used as a classification label of multi-view data. According to the method, the accuracy of multi-view data classification is integrally improved, and the method can be applied to various fields including multi-view data, such as computer vision and natural language analysis.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and multi-view learning, and in particular to a multi-view data category recognition method based on contrastive learning and private information. Background Art

[0002] In today's information age, data collection methods are becoming increasingly diverse. From traditional questionnaires and sensor monitoring to modern social media mining and IoT device perception, the sources and types of data are showing unprecedented diversity. This diversity is reflected not only in the surge in data volume, but also in the diversification of data perspectives or "views." For example, a product can be represented through multiple methods such as images, text descriptions, and user reviews; a scientific research project may include multi-dimensional information such as research papers, patent documents, and project reports; and a webpage can be represented using its content text and hyperlinks pointing to the page. This multi-view data provides more comprehensive and detailed features for describing objects, helping computer models to discover deeper data connections and patterns.

[0003] Multi-view learning, a key branch of machine learning, is a method for effectively integrating and analyzing this multi-source, multi-modal data. Generally speaking, data from different views contains both consistent and complementary information. The former means that data from multiple views will share some common features of the samples, while the latter indicates that different views also contain their own unique private information. This private information is generally generated by differences in observation, collection, and data processing methods. Multi-view learning leverages the multiple information between data from different views and achieves information fusion and complementarity through algorithmic design, thereby improving model performance and generalization. Therefore, compared to a single view, multi-view learning provides a more comprehensive representation, resulting in superior model results.

[0004] However, as multi-view learning research deepens, a series of challenges have gradually emerged. While existing multi-view learning methods have achieved significant results, they primarily focus on extracting common information that represents consistency, assuming that private information from different views may interfere with cross-view feature fusion, resulting in relatively limited use of intra-view private information. However, as the number of views increases, information conflicts between views also increase, resulting in a gradual decrease in the common information extracted from multiple views, leading to poor representation. Furthermore, the private information of each view represents the characteristics of the scene in that view. While this information may affect cross-view information fusion, its contribution to supplementing the information within the view itself cannot be ignored. This private information can provide more basis for intra-view feature learning. Therefore, how to effectively utilize this private information is a major challenge facing multi-view learning. Furthermore, after obtaining features learned from private information, how to connect them with the original public information is also a question worth exploring. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention proposes a multi-view data category recognition method based on contrastive learning and private information. Private information learning is performed on multi-view data to achieve comparison of close neighbors within the view and public semantic information learning is performed to compare the same instance labels across views. Finally, consistency constraints are performed through clustering, and the semantic labels are used as the final classification results.

[0006] The present invention implements a multi-view data category recognition method based on contrastive learning and private information, comprising the following steps:

[0007] S1. Use autoencoders to obtain mixed features from multiple views of the input;

[0008] S2. Based on the hybrid features, a private feature set and a public semantic feature set are obtained. The private feature set is used as the object for same-view instance-neighbor private feature learning to obtain useful private features. The public semantic feature set is used for cross-view semantic information consistency feature learning to obtain aligned public semantic features.

[0009] S3. Introduce the K-Means clustering prediction model to generate pseudo labels for the useful private features, perform corresponding constraints based on the pseudo labels of the private features and the predicted labels of the aligned public semantic features, and use the final label with the largest average value among all views as the classification label for the multi-view data.

[0010] S1 also includes constructing an autoencoder corresponding to each view, using the encoder to map the initial features of the data of each view into mixed features, and then using the decoder corresponding to each view to reconstruct the original mixed features to achieve mixed feature optimization.

[0011] In some embodiments, S2 also includes performing instance-based neighbor private feature learning in the same view with the private feature set as the object, further including: calculating the distance between each instance included in each view, finding the neighbor instance closest to each instance, and based on comparative learning, obtaining useful private features in multiple views by bringing similar instances closer and pushing dissimilar instances further away.

[0012] In some embodiments, S3 further includes obtaining instances with the same index from multiple views based on the index of the instance to form cross-view instance pairs. Based on contrastive learning, consistency constraints are imposed on these cross-view instance pairs so that the same instance maintains the same semantic information in multiple views.

[0013] In some embodiments, introducing a K-Means clustering prediction model to generate pseudo labels of the useful private features further includes the K-Means clustering prediction model taking the learned private feature set H as input, and for any private feature H of a view v with N instances, v , if the data set has C categories, then use the clustering algorithm to cluster H v The N vectors are clustered into C clusters and an N×C dimensional vector is output. Each instance has a 1×C dimensional one-hot vector. The position index with a value of 1 in this vector represents the predicted category of the instance, which is output as a private feature pseudo-label.

[0014] In some embodiments, S1 further includes all view original reconstruction losses

[0015]

[0016] in, represents the sum of all view reconstruction losses, represents the reconstruction loss of the mth view, m represents the view index, N represents the number of instances, Represents a composite mapping of two functions, R N Represents N-dimensional real feature space;

[0017] In some embodiments, in S2, the same view instance neighbor private feature learning further includes private information contrast loss

[0018]

[0019] Denote the contrastive loss of private information per view based on contrastive learning:

[0020]

[0021] Among them, τ H The temperature coefficient representing the private information part, Representation instance and its mth neighbor The cosine distance, Indicates the distance between two instances.

[0022] In some embodiments, in S2, the same view instance-based neighbor private feature learning further includes a public semantic contrast loss

[0023]

[0024] Represents the common semantic contrast loss between any two views

[0025]

[0026] Among them, τ P is the temperature coefficient of the public semantic information part, express When v1=v a The common semantic contrast loss between all two different views of and, Indicates that v1 takes all values from 1 to V The harmony.

[0027] In some embodiments, in S3, private information and public semantic constraints cross entropy loss for:

[0028]

[0029] in, Indicates that the cluster order of private features is rearranged to obtain a new private feature clustering result, P v Indicates common semantic features.

[0030] In some embodiments, the final loss function of the method is

[0031]

[0032] Among them, λ represents the hyperparameter used to balance the contrast loss and the constraint loss, represents the autoencoder reconstruction loss, represents the private contrast loss, represents the public semantic contrast loss, Represents private information and public semantically constrained cross entropy loss.

[0033] Compared with the prior art, the present invention can achieve the following beneficial technical effects:

[0034] 1) A multi-view learning method based on contrastive learning that simultaneously uses private information to constrain the consistency of cross-view data based on private features of instance neighbors in the same view and public information. This method emphasizes the use of special private information in each view, so that the characteristics of multi-view learning include more view-specific context information, and constrains the learned results with the original public semantic information to achieve better learning effect;

[0035] 2) During the learning process, the original reconstruction loss is designed Private Information Department Comparison Loss Public semantics versus loss of private information and public semantically constrained cross entropy Loss and the final loss function constructed To achieve parameter optimization of the neural network learning process;

[0036] 2) The accuracy of multi-view data classification is improved overall, and it can be applied to various fields involving multi-view data, such as computer vision and natural language analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of a flow chart of a multi-view data category recognition method based on contrastive learning and private information according to the present invention;

[0038] Figure 2 This is a framework diagram for implementing the technology of the present invention. DETAILED DESCRIPTION

[0039] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0040] This paper provides a multi-view data classification method based on contrastive learning and private information. It uses an autoencoder to process raw data to obtain hybrid features. It then processes the private and public information of each instance separately. Finally, it uses the naturally generated clusters of private and public semantic information to apply consistency constraints to link the two types of information. This method then uses clustering to classify the multi-view data. This method can be widely applied to image recognition, data analysis, and other fields.

[0041] like Figure 1As shown, the overall process of the multi-view data category recognition method based on contrastive learning and private information of the present invention includes the following steps:

[0042] Step 1: Input multiple views as initial data, each view including instances, construct an autoencoder for each view, and perform mixed feature extraction on the data of multiple views through the autoencoder, including using the encoder to map the initial features of the data of each view into mixed features, and then reconstructing the original mixed features through v decoders to achieve mixed feature optimization;

[0043] Step 2: Use two linear perceptrons to separate the private information and public semantic information of the mixed features, and obtain the private feature set and public semantic feature set of the mixed features respectively. That is, use two linear perceptrons to learn the private features and public semantic features from the mixed feature set respectively, that is, Z v Mapped into H through two perceptrons respectively v and P v , thereby strengthening the application of two types of information;

[0044] The public semantic information refers to multiple views of a data instance. Since different views describe the same instance, different views will have public semantic information for the same instance. The private information refers to the different ways of obtaining data in different views. Therefore, for the same instance, the representation in different views contains private information representing the view environment. That is, the information unique to a single view is private information, and the information common to all views is public information. 2. For all instances in any view v, the mixed representation Z v , use two perceptrons to perform mapping calculations to obtain feature representation H v and P v , the former subsequently participates in the feature processing of private information, and the latter participates in the feature processing of public information, so that H v and P v Represents private information and public information respectively.

[0045] ① Private feature learning: Using same-view instance-based neighbor private feature learning. This process involves calculating the distance between each instance in each view, finding the k nearest neighbor instances to each instance, and then using contrastive learning to bring similar instances closer and push dissimilar instances further away, thereby obtaining useful private features within multiple views.

[0046] ② Common semantic feature learning is performed using cross-view semantic information consistency feature learning. This process involves obtaining instances with the same index from multiple views based on their index, forming cross-view instance pairs, and then applying consistency constraints to these cross-view instance pairs based on contrastive learning, so that the same instance maintains the same semantic information in multiple views, achieving common semantic feature alignment.

[0047] Step 3: After completing the learning of the private feature set and the public semantic feature set of the multi-view data, use the K-Means clustering model to predict and generate pseudo labels for all private features, and constrain these pseudo labels to correspond with the predicted labels of the public semantic features. Use the private features to improve the classification effect of the public semantic features. After the corresponding constraints are completed, the final label with the largest average predicted probability among all views is used as the classification label of the multi-view data.

[0048] Among them, for the K-Means clustering model, the input is the learned private feature set H, for any private feature H of a view v with N instances v , if the data set has C categories, then use the clustering algorithm to cluster H v The N vectors are clustered into C clusters and an N×C dimensional vector is output. That is, each instance has a 1×C dimensional one-hot vector. The position index with a value of 1 in this vector represents the predicted category of the instance, which is output as a private feature pseudo-label.

[0049] Specifically, the predicted labels of the public semantic features are separated by the perceptron to obtain the public information P v The dimension is N×C, where C is the category in the data set, so P v It can directly represent the predicted label of each instance.

[0050] like Figure 2 The multi-view learning architecture based on contrastive learning of the method of the present invention is shown as follows: Represents V view original data, and V mixed features are learned through multiple encoders Then it is separated into private information through two MLPs and public semantic information H uses the nearest neighbors within the view for comparison, P uses the same instance labels across views for comparison, and finally performs consistency constraints through clustering, and uses the semantic label as the final classification result.

[0051] Specifically, in step 1, for a model with V views, each view has N instances, and the instance dimension is D v A multi-view dataset with C data categories Build an encoder f for each view vand decoder g v , using f v The original data set X v Convert to d v dimensional mixed feature set Z v ,Right now Then, using the decoder g v Perform original reconstruction on the mixed feature set to obtain the reconstructed dataset Right now The original reconstruction loss for all V views is:

[0052]

[0053] in, represents the sum of all view reconstruction losses, represents the reconstruction loss of the mth view, m represents the view index, i.e. 1, 2, ..., v, and N represents the number of instances, i.e. there are N instances or objects in a view. Represents a composite mapping of two functions, such as R N Represents the N-dimensional real feature space.

[0054] Specifically, in step 2, first use two linear perceptrons to extract the mixed feature set Z v Private feature sets are learned in and public semantic feature sets

[0055] In the process of private feature learning, the same view is used to learn the private features based on the instance neighbors. v Each instance in The k neighbors of Right now:

[0056]

[0057] in, represents the distance between an instance and its neighbor instances, Indicates taking the k neighbors with the smallest distance to the instance. In the feature space of There are N-1 feature pairs, k neighbor features are used as positive pairs, and the remaining Nk features are used as negative pairs. The cosine distance is used to measure the similarity of the feature pairs and calculate the cosine distance. for:

[0058]

[0059] in, represents the i-th private feature of the v-th view, represents the jth private feature of the vth view, i≠j, that is, i and j are any two private features in the vth view;

[0060] Then the contrastive loss of each view private information based on contrastive learning is:

[0061]

[0062] Among them, τ H The temperature coefficient representing the private information part, Representation instance and its mth neighbor The cosine distance is taken as a positive value using exp and divided by the temperature coefficient to ensure it is within a reasonable range; Indicates the distance value when two instances are the same. Subtract this variable to remove the distance between the instance and itself.

[0063] The private information contrast loss of all views is summed up as the total private information contrast loss in the entire learning process.

[0064]

[0065] In the process of learning common semantic information, a cross-view semantic information consistency feature learning process is adopted. Each instance has VN-1 common semantic feature pairs in all views, that is, And v a ≠v b , where v a With v b Indicates the index of the view, i.e. the vth a Views and v b Views; Indicates the vth a The common semantic vector of the i-th instance in the view, Similarly, j and N have the same meaning as above, i.e., j represents any index, N represents the number of instances in the dataset, V-1 instance pairs corresponding to the same instance are regarded as semantic positive pairs, and the remaining V(N-1) instance pairs are regarded as semantic negative pairs. Then the common semantic contrast loss between any two views is for:

[0066]

[0067] Among them, τ P is the temperature coefficient of the public semantic information part.

[0068] Common semantic contrast loss across multiple views for:

[0069]

[0070] in, express When v1=v a All different situations <v a ≠v b >The common semantic contrast loss between the two views of and, Indicates that v1 can take all values from 1 to V The harmony.

[0071] Specifically, in step 3, the K-means algorithm is used to obtain the clustering results of the private features of each view. in, Represents the one-hot encoding of each instance cluster label, i.e. In the C-dimensional vector represented by , only the index position of its label is 1. The Hungarian algorithm is further used to rearrange the cluster order of private features to obtain a new private feature clustering result representation Make the clusters of the new private feature clustering results correspond one-to-one with the clusters of public semantic features, constrain both private information and public semantics, and constrain the cross entropy loss between private information and public semantics. for:

[0072]

[0073] Among them, P v Represents the public semantic features and is also the probability vector of the final semantic label;

[0074] The final loss function Sum the four partial losses:

[0075]

[0076] Among them, λ represents the hyperparameter used to balance the contrast loss and the constraint loss, represents the autoencoder reconstruction loss, represents the private contrast loss, represents the public semantic contrast loss, Represents private information and public semantically constrained cross entropy loss.

[0077] The public semantic feature P v The final label is the classification label of the multi-view data, and the data category result y of the i-th instance i for:

[0078]

[0079] in, The final feature label representing the public semantics of the i-th instance of the v-th view, It represents the label index with the largest average final predicted probability of the common semantic features in all V views.

[0080] For example, if there is a data set with 3 views and 4 categories, then for any instance i, there are 3 4-dimensional public semantic prediction label probability vectors, such as [0.2, 0.6, 0.1, 0.1], [0.1, 0.7, 0.15, 0.05], [0.05, 0.8, 0.02, 0.08], which represent Sum and average them to get [0.12, 0.7, 0.09, 0.09]. The second position is the largest, indicating that the predicted probability of the second label is the largest. (Here we assume that the index starts from 1), that is, y i =2).

[0081] The above exemplary embodiments illustrate and describe the overall technical solution of the present invention, but those skilled in the art should understand that the present invention is not limited to the above-mentioned embodiments, and all changes or replacements to the present invention fall within the scope of protection of the present invention.

Claims

1. A multi-view data category recognition method based on contrastive learning and private information, characterized in that: The following steps are involved: S1. Use autoencoders to obtain mixed features from multiple views of the input; S2. Based on the hybrid features, a private feature set and a public semantic feature set are obtained. The private feature set is used as the object for same-view instance-neighbor private feature learning to obtain useful private features. The public semantic feature set is used for cross-view semantic information consistency feature learning to obtain aligned public semantic features. S3. Introduce the K-Means clustering prediction model to generate pseudo labels for the useful private features, perform corresponding constraints based on the pseudo labels of the private features and the predicted labels of the aligned public semantic features, and use the final label with the largest average predicted probability among all views as the classification label for the multi-view data.

2. The multi-view data category recognition method based on contrastive learning and private information according to claim 1, characterized in that: S1 also includes constructing an autoencoder corresponding to each view, using the encoder to map the initial features of the data of each view into mixed features, and then using the decoder corresponding to each view to reconstruct the original mixed features to achieve mixed feature optimization.

3. The multi-view data category recognition method based on contrastive learning and private information according to claim 1, characterized in that: S2 also includes the same-view instance-neighbor private feature learning based on the private feature set, further including: calculating the distance between each instance included in each view, finding the neighbor instance closest to each instance, and based on comparative learning, obtaining useful private features in multiple views by bringing similar instances closer and pushing dissimilar instances further away.

4. The multi-view data category recognition method based on contrastive learning and private information according to claim 1, characterized in that: S3 also includes obtaining instances with the same index from multiple views based on the instance index to form cross-view instance pairs. Based on contrastive learning, consistency constraints are imposed on these cross-view instance pairs so that the same instance maintains the same semantic information in multiple views.

5. The multi-view data category recognition method based on contrastive learning and private information according to claim 1, characterized in that: Introducing the K-Means clustering prediction model to generate the pseudo-label of the useful private feature further includes the K-Means clustering prediction model taking the learned private feature set H as input, and for any private feature H of a view v with N instances v , if the data set has C categories, then use the clustering algorithm to cluster H v The N vectors are clustered into C clusters and an N×C dimensional vector is output. Each instance has a 1×C dimensional one-hot vector. The position index with a value of 1 in this vector represents the predicted category of the instance, which is output as a private feature pseudo-label.

6. The multi-view data category recognition method based on contrastive learning and private information according to claim 2, characterized in that: S1 further includes the original reconstruction loss of all views in, represents the sum of all view reconstruction losses, represents the reconstruction loss of the mth view, m represents the view index, N represents the number of instances, Represents a composite mapping of two functions, R N Represents N-dimensional real feature space; 7. The multi-view data category recognition method based on contrastive learning and private information according to claim 5, characterized in that: In S2, the same view is further included in the contrast loss based on the private feature learning of instance neighbors. Denote the contrastive loss of private information per view based on contrastive learning: Among them, τ H The temperature coefficient representing the private information part, Representation instance and its mth neighbor The cosine distance, Indicates the distance between two instances.

8. The multi-view data category recognition method based on contrastive learning and private information according to claim 1, characterized in that: In S2, the same view further includes public semantic contrast loss based on instance neighbor private feature learning Represents the common semantic contrast loss between any two views Among them, τ P is the temperature coefficient of the public semantic information part, express When v1=v a The common semantic contrast loss between all two different views of and, Indicates that v1 takes all values from 1 to V The harmony.

9. The multi-view data category recognition method based on contrastive learning and private information according to claim 1, characterized in that: In S3, private information and public semantic constraints cross entropy loss for: in, Indicates that the cluster order of private features is rearranged to obtain a new private feature clustering result, P v Indicates common semantic features.

10. The multi-view data category recognition method based on contrastive learning and private information according to claim 1, characterized in that: The final loss function of this method is Among them, λ represents the hyperparameter used to balance the contrast loss and the constraint loss, represents the autoencoder reconstruction loss, represents the private contrast loss, represents the public semantic contrast loss, Represents private information and public semantically constrained cross entropy loss.

Images