Multi-view clustering method and device based on hierarchical prompt guide alignment

Abstract

The invention provides a multi-view clustering method and device based on hierarchical prompt guide alignment, and the method comprises the steps: obtaining to-be-clustered multi-view data for a target scene, constructing a main layer structure, and generating a preliminary semantic feature representation; an embedded layer structure is constructed, local and global high-level semantic representations are obtained through local and global prompt generation networks and a double-mapping encoder, and an attention mechanism is introduced to fuse the local high-level semantic representations of all modes to obtain a fused semantic representation; constructing a label layer, generating a plurality of pseudo labels by using a pseudo label generation module, and selecting a maximum value for normalization to obtain a global clustering label; and finally, carrying out parameter updating on related structures and modules by adopting comparative learning and the like to construct joint loss so as to generate a clustering result. According to the method, effective information of each modal of multi-view data is fully mined and deeply fused through hierarchical prompt guidance and multi-level alignment operation, so that the accuracy and robustness of multi-view clustering are effectively improved, and a clustering result can more accurately reflect a real semantic structure of the data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-view data processing, and in particular to a multi-view clustering method and device based on hierarchical prompt guided alignment. BACKGROUND

[0002] Although the current multi-view clustering technology is widely used in image retrieval, social network analysis and other fields, it has obvious limitations. Traditional methods such as subspace assumption or tensor decomposition-based models often rely on linear assumptions or tensor decomposition models when integrating different view information, making it difficult to effectively handle high-dimensional, nonlinear features, resulting in limited performance in complex data environments. For example, in the intelligent manufacturing scenario, different types of sensor data have significant differences, and traditional methods are difficult to simultaneously depict the fine-grained dynamics of equipment health evolution and cross-view consistency, resulting in delayed abnormal cluster identification or high false alarm rates. Figure One

[0003] In addition, the existing technology has difficulty in effectively aligning the distribution differences between the feature layer and the clustering result layer. Some methods focus on global distribution alignment or subspace consistency between views, but ignore the balance between "fine-grained semantics" within the view and "local differences" between views, making it difficult to fully exploit the unique information of each view. In terms of prompt learning, although it has shown strong guiding ability in natural language processing and computer vision, the application of prompt learning to multi-view clustering tasks is still in its early stages, especially the lack of systematic methods to design hierarchical prompt structures to capture fine-grained semantics and shared features.

[0004] The existing technology also has difficulty in balancing global consistency and local differences between views. Some methods only focus on global alignment, which can easily lead to loss of view detail information, while others introduce local feature enhancement but fail to effectively establish global consistency between views, affecting the stability and accuracy of the clustering results. Therefore, there is an urgent need for a new solution to handle multi-view clustering tasks. SUMMARY

[0005] In view of this, the embodiments of the present application provide a multi-view clustering method and device based on hierarchical prompt guided alignment to eliminate or improve one or more defects in the prior art, and solve the problem that the prior art cannot effectively balance fine-grained semantic mining within the view and global consistency modeling between views in multi-view clustering.

[0006] In one aspect, the present application provides a multi-view clustering method based on hierarchical prompt guided alignment, which comprises the following steps: Obtain multi-view data to be clustered for a target scene, the multi-view data containing a plurality of samples, and the samples having a plurality of modal views;​ constructing a main layer structure, adopting a main layer prompt encoder to take the sample as input and output a main layer level prompt vector of each modality, adding the main layer level prompt vector of each modality to original data and inputting a shallow layer encoder to obtain a preliminary semantic feature representation of each modality; constructing an embedding layer structure, respectively establishing a local prompt generation network for each modality to take the preliminary semantic feature representation of the corresponding modality as input and output a local prompt, establishing a global prompt generation network to take the fused preliminary semantic feature representation based on adaptive weight parameters as input and output a global prompt, and projecting the fused preliminary semantic feature representation of each modality in the sample, the local prompt and the global prompt to a semantic clustering space through a first mapping encoder to obtain a corresponding local high-level semantic representation; projecting the global prompt to the semantic clustering space through a second mapping encoder to obtain a corresponding global high-level semantic representation; introducing an attention mechanism to fuse the local high-level semantic representation of each modality of the sample to obtain a fused semantic representation; constructing a label layer, inputting the local high-level semantic representation of each modality of the sample into a representation-level pseudo label generation module to obtain a local pseudo label corresponding to each modality, inputting the fused semantic representation into the representation-level pseudo label generation module to obtain a fused pseudo label, inputting the global high-level semantic representation into a prompt-level pseudo label generation module to obtain a global pseudo label, and selecting a maximum value in the local pseudo label, the fused pseudo label and the global pseudo label for each classification category to obtain a global clustering label after normalization; adopting contrast learning to perform alignment operation on the embedding layer structure and the label layer to establish a contrast loss; calculating a distribution probability difference between the global clustering label and the fused pseudo label and suppressing information entropy of the local pseudo label and the global pseudo label to construct a forward alignment loss; adopting a transition mapping module to perform transition mapping on the local pseudo label and the global pseudo label, and calculating a distribution probability difference between the original global pseudo label and the global clustering label to construct a reverse feedback loss; based on the contrast loss, the forward alignment loss and the reverse feedback loss, constructing a joint loss to update parameters of the embedding layer structure, the label layer and the transition mapping module, and generating a clustering result based on the global clustering label obtained after updating.

[0007] In some embodiments, the main layer prompt encoder adopts a multi-layer perception encoder, the shallow layer encoder adopts a multi-layer perception encoder or a graph convolution encoder; the representation-level pseudo label generation module and the prompt-level pseudo label generation module adopt a nonlinear support vector machine with the same structure.

[0008] In some embodiments, the global prompt generation network is established to take the fused preliminary semantic feature representation based on adaptive weight parameters as input and output a global prompt, and the calculation formula is: ; wherein, denotes the global cue, denotes the global cue generation network, denotes an adaptive weight parameter of the vth modality data of the sample, denotes the preliminary semantic feature representation of the vth modality data of the sample.

[0009] In some embodiments, the preliminary semantic feature representation corresponding to each modality in the sample, the local cue and the global cue are fused and projected to a semantic clustering space by a first mapping encoder to obtain a corresponding local high-level semantic representation, and the calculation formula is as follows: ; wherein, denotes the local high-level semantic representation corresponding to the vth modality data of the sample, denotes the first mapping encoder, denotes the vth modality data of the sample, is a preset parameter, denotes the local cue corresponding to the vth modality data of the sample.

[0010] In some embodiments, the local high-level semantic representation of each modality of the sample is introduced into an attention mechanism for fusion to obtain a fused semantic representation, including: a weight coefficient is calculated based on an attention mechanism network and a nonlinear feedforward neural network, and the calculation formula is as follows: ; the local high-level semantic representation corresponding to the v modality data in the sample is fused based on the weight coefficient, and the calculation formula is as follows: ; wherein, denotes element-wise multiplication, denotes a weight matrix of the vth modality data in the sample, denotes the attention mechanism network, denotes the nonlinear feedforward neural network, denotes the fused semantic representation.

[0011] In some embodiments, contrast learning is used to align the embedding layer structure and the label layer to establish a contrast loss, and the calculation formula is as follows: ; wherein, denotes the contrast loss, denotes the local high-level semantic representation corresponding to the k-th modality data of the i-th sample, denotes the modality category number, and m and n denote the sample number; denotes the label matrix of each sample in the m-th category corresponding to the k-th modality, m and n denote the category number, N denotes the sample number, and C denotes the category number; denotes the cosine similarity function.

[0012] In some embodiments, the maximum value in the local pseudo label, the fusion pseudo label and the global pseudo label is selected for each classification category to obtain a global clustering label by normalization, comprising: The maximum value of the label is selected in the local pseudo label, the fusion pseudo label and the global pseudo label for the i-th sample about the m-th classification, and the expression is: ; wherein, denotes the global pseudo label of the i-th sample about the m-th classification, denotes the local pseudo label of the v-th modality in the i-th sample about the m-th classification, denotes the fusion pseudo label of the i-th sample about the m-th classification; The global clustering label of a single sample is calculated, and the calculation formula is: ; wherein, denotes the value corresponding to the p-th classification in the global clustering label, denotes the maximum value of the label about the p-th classification, denotes the maximum value of the label about the q-th classification, and C is the number of classifications; The calculation formula of the forward alignment loss is: ; wherein, denotes the forward alignment loss, denotes the local pseudo label matrix of each sample about the modality v, denotes the global pseudo label matrix, denotes the global clustering label matrix, denotes the fusion pseudo label matrix; denotes the KL divergence; denotes the information entropy.

[0013] In some embodiments, the local pseudo label and the global pseudo label are transition mapped by using a transition mapping module, and the expression is: ; ; wherein, represents a ReLU activation function, represents an intermediate representation learned from the global pseudo label matrix , represents an intermediate representation learned from the fusion pseudo label matrix ; , , and are weight matrices; , , and are fixed values; The calculation formula of the back feedback loss is: ; wherein, represents the back feedback loss; The calculation formula of the joint loss is: ; wherein, is the joint loss, and are weights.

[0014] In another aspect, the present application also provides a multi-view clustering method device based on hierarchical prompt guidance alignment, comprising a processor, a memory and computer programs / instructions stored on the memory, wherein the processor is used to execute the computer programs / instructions, and when the computer programs / instructions are executed, the device realizes the steps of the above method.

[0015] In another aspect, the present application also provides a computer readable storage medium, which stores computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the above method are realized.

[0016] The multi-view clustering method and device based on hierarchical prompt guided alignment provided by the present application construct a main layer structure and an embedded layer structure to extract semantic features of multi-modal data, and optimize clustering results through a label layer and a bidirectional alignment mechanism. The main layer prompt encoder generates a main layer level prompt vector for each modality and adds it to the original data, and the preliminary semantic feature representation is obtained through a shallow layer encoder, which can enhance the original input data, retain and mine the unique semantic information of each view. The local prompt generation network and the global prompt generation network in the embedded layer structure output local prompts and global prompts respectively, and after fusion, the mapping encoder is used to project to the semantic clustering space to obtain local and global high-level semantic representations, and the attention mechanism is used to fuse the local high-level semantic representations of each modality, realizing fine-grained semantic enhancement and cross-view semantic alignment optimization. The label layer generates local pseudo labels, fused pseudo labels and global pseudo labels of each modality, and the global clustering label is obtained by selecting the maximum value and normalizing, which strengthens the most discriminative signal and weakens the interference of low confidence views. The contrast loss of the embedded layer structure and the label layer is established through contrast learning, the forward alignment loss is constructed by combining the distribution probability difference between the global clustering label and the fused pseudo label, and the reverse feedback loss is constructed by using the transition mapping module. Finally, the parameters are updated based on the joint loss to generate the clustering result, which takes into account the fine-grained semantic mining within the view and the global consistency between the views, and improves the clustering accuracy and stability.

[0017] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the specification and claims hereof as well as the appended drawings.

[0018] It will be understood by those skilled in the art that the objects and advantages of the present application can be realized and attained by the structure particularly pointed out in the specification and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings: Figure 1 The flowchart of the multi-view clustering method based on hierarchical prompt guided alignment according to an embodiment of the present application.

[0020] Figure 2 The logic structure diagram of the multi-view clustering method based on hierarchical prompt guided alignment according to another embodiment of the present application.

[0021] Figure 3FIG. 1 is a sensitivity diagram of different values of superparameters β and γ in joint loss based on Caltech7 data according to an embodiment of the present application.

[0022] Figure 4 FIG. 2 is a sensitivity diagram of different values of superparameters β and γ in joint loss based on Citeseer data according to an embodiment of the present application.

[0023] Figure 5 FIG. 3 is a diagram of original features of BBCSport dataset.

[0024] Figure 6 FIG. 4 is a clustering effect diagram of BBCSport dataset in HiPMVC method according to an embodiment of the present application. Figure 5

[0025] Figure 7 FIG. 5 is a diagram of original features of Caltech7 dataset.

[0026] Figure 8 FIG. 6 is a clustering effect diagram of Caltech7 dataset in HiPMVC method according to an embodiment of the present application. Figure 7 DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with embodiments and drawings. Herein, the illustrative embodiments of the present application and the description thereof are used to explain the present application, but not as a limitation to the present application. Herein, it should also be noted that, in order to avoid the present application being obscured by unnecessary details, only structures and / or processing steps closely related to the solution according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0028] It should be emphasized that the term “comprises / comprising” as used herein indicates the presence of the stated features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0029] In a large-scale, multi-source data environment, the information complementarity of different modalities or different feature views provides rich support for clustering tasks, and multi-view clustering (MVC) has therefore been widely concerned in the fields of image retrieval, social network analysis, bioinformatics, etc. Traditional multi-view clustering methods are mostly based on subspace assumption or tensor decomposition, and the shared subspace or low-rank tensor representation is found in each view feature space to realize collaborative learning between views. Although these methods can integrate the complementary information of different views to a certain extent, they often rely on linear assumption or tensor decomposition model, and the performance will be limited when facing high-dimensional, nonlinear features.​​

[0030] Multi-view clustering is an unsupervised learning method for processing multi-source and multi-modal data. In many practical scenarios, data objects can be described from multiple perspectives or modalities. These perspectives can be different types of data sources (such as images, text, audio, etc.), or different feature representations of the same data. The goal of multi-view clustering is to integrate data from different perspectives, mine the inherent structure and pattern of the data, and divide similar data objects into the same cluster. Traditional clustering methods usually assume that data samples have only one feature representation, while multi-view clustering considers the multi-modality of data and can comprehensively utilize information from multiple perspectives to obtain more comprehensive and accurate data division results. The present application introduces prompt learning, which is a learning paradigm that guides the model to complete downstream tasks. By injecting a learnable prompt vector into the input, the existing knowledge in the pre-trained model is activated, and the amount of parameter updates is reduced. It has been widely applied in natural language processing and computer vision fields. The core idea is to introduce additional information at the input end to guide the model to focus on task-related features and achieve efficient representation learning.

[0031] The present application provides a multi-view clustering method based on hierarchical prompt guided alignment, as shown in Figure 1 The method comprises the following steps S101-S105: Step S101: Obtain multi-view data to be clustered for a target scene, the multi-view data comprising a plurality of samples, and the samples having a plurality of modal views.

[0032] Step S102: Construct a main layer structure, use a main layer prompt encoder to input the samples and output a main layer prompt vector of each modality, add the main layer prompt vector of each modality to the original data and input a shallow layer encoder to obtain a preliminary semantic feature representation of each modality.

[0033] Step S103: Construct an embedding layer structure, establish a local prompt generation network for each modality to input the preliminary semantic feature representation of the corresponding modality and output a local prompt; establish a global prompt generation network to input the fused preliminary semantic feature representation based on adaptive weight parameters and output a global prompt; fuse the preliminary semantic feature representation, the local prompt and the global prompt of each modality in the sample through a first mapping encoder to project to a semantic clustering space to obtain a corresponding local high-level semantic representation; project the global prompt through a second mapping encoder to obtain a corresponding global high-level semantic representation; introduce an attention mechanism to fuse the local high-level semantic representations of each modality of the sample to obtain a fused semantic representation.

[0034] Step S104: constructing a label layer, inputting the local high-level semantic representation of each modality of the sample into a representation-level pseudo-label generation module to obtain the local pseudo-label corresponding to each modality, inputting the fused semantic representation into the representation-level pseudo-label generation module to obtain the fused pseudo-label, inputting the global high-level semantic representation into a prompt-level pseudo-label generation module to obtain the global pseudo-label, and selecting the maximum value in the local pseudo-label, the fused pseudo-label and the global pseudo-label for each classification category to obtain a global clustering label after normalization.

[0035] Step S105: adopting contrast learning to perform alignment operation on the embedding layer structure and the label layer to establish a contrast loss, calculating the distribution probability difference between the global clustering label and the fused pseudo-label and suppressing the information entropy of the local pseudo-label and the global pseudo-label to construct a forward alignment loss, adopting a transition mapping module to perform transition mapping on the local pseudo-label and the global pseudo-label, and calculating the distribution probability difference with the original global pseudo-label and the global clustering label to construct a reverse feedback loss, and based on the contrast loss, the forward alignment loss and the reverse feedback loss, constructing a joint loss to update the parameters of the embedding layer structure, the label layer and the transition mapping module, and generating a clustering result based on the global clustering label obtained after the update.

[0036] In step S101, the multi-view data to be clustered for the target scene contains multiple samples, and each sample has multiple modal views. For example, in the intelligent manufacturing scene, various heterogeneous sensors such as vibration, temperature, current and acoustic sensors are arranged on the production line to generate high-speed streaming data with different sampling frequencies, and these different modal data constitute multi-view data. The multi-view data can be represented as wherein, represents the number of samples, represents the number of modalities, is the data dimension of the th modality.

[0037] In step S102, a main layer structure is constructed, a main layer prompt encoder is adopted to input each modality data of each sample as input and output a main layer prompt vector of each modality, the main layer prompt vector of each modality is added to the original data and input into a shallow layer encoder to obtain a preliminary semantic feature representation of each modality. This process generates a prompt vector for the data of each modality through the main layer prompt encoder, aiming to enhance the original input data, retain and mine the unique semantic information of each view.

[0038] The master-level cue encoder generates modality-adaptive master-level cue vectors for each modality view in the sample. By adding the cue vectors element-wise to the original data, the shallow encoder is forced to prioritize local features strongly correlated with the clustering task, suppressing irrelevant noise. Compared to traditional subspace methods that force views to share the same linear space, this application preserves the physical characteristics of each view before feature mapping. For example, in bearing monitoring, the high-frequency resonance component (5-10kHz) of the vibration signal and the slow variation trend of the temperature signal are enhanced separately, avoiding the blurring of heterogeneous features in early fusion. The output features serve as input for subsequent processing, and their cluster separability is significantly improved through cue guidance.

[0039] In some embodiments, the main layer prompt encoder is a multilayer perceptron encoder, and the shallow layer encoder is a multilayer perceptron encoder or a graph convolution encoder.

[0040] In some embodiments, a global prompt generation network is established to take the preliminary semantic features fused based on adaptive weight parameters as input and output global prompts, and the calculation formula is as follows: ; in, This indicates a global suggestion. This indicates that a global suggestion is generated for the network. This represents the adaptive weight parameters for the v-th modality of the sample. This represents the preliminary semantic feature representation of the v-th modality data of the sample.

[0041] In step S103, a local-global dual-cue collaboration mechanism is used to achieve fine-grained view semantics and cross-view functionality during the feature fusion stage. Figure One The dynamic balancing of consistency fundamentally solves the problem of complementary information loss caused by coarse-grained fusion in traditional methods. Specifically, fine-grained local semantic enhancement is performed, and the local cue generation network extracts view-specific clustering cues from the preliminary semantic feature representations of each view. Features are projected into a more discriminative subspace through weighted fusion and a mapping encoder. Cross-view semantic alignment optimization is performed, and global cues dynamically learn the correlation patterns between views by adaptively fusion of multi-view features.

[0042] In some embodiments, the preliminary semantic feature representation, local cue, and global cue corresponding to each modality in the sample are fused and projected onto the semantic clustering space through a first mapping encoder to obtain the corresponding local high-level semantic representation, as calculated below: ; in, This represents the local high-level semantic representation corresponding to the v-th modality of the sample. Indicates the first mapping encoder. This represents the v-th modal data of the sample. These are preset parameters. This indicates the local cue corresponding to the vth modal data of the sample.

[0043] In some embodiments, the local high-level semantic representations of each modality of the sample are fused using an attention mechanism to obtain a fused semantic representation, including: The weight coefficients are calculated based on attention mechanism networks and nonlinear feedforward neural networks, using the following formula: ; The local high-level semantic representations corresponding to v modal data in the sample are fused based on weighting coefficients, and the calculation formula is as follows: ; in, This represents the weight matrix for the v-th modality in the sample. This represents an attention mechanism network. This represents a nonlinear feedforward neural network. This represents the fused semantic representation.

[0044] In step S104, this step achieves fine-grained view cues and cross-view functionality at the label level through three-layer pseudo-label generation and maximum value normalization aggregation. Figure One The deep integration of consistency-based decision-making effectively solves the misjudgment problem caused by label ambiguity or view conflict in traditional methods. Specifically, hierarchical labels exhibit semantic complementarity, local pseudo-labels mine fine-grained features within views, fused pseudo-labels integrate multiple views through an attention mechanism, and global pseudo-labels are generated based on shared semantics across views. By taking the maximum value of local / fused / global labels and normalizing it to generate global cluster labels, the most discriminative signals are strengthened, while interference from low-confidence views is weakened. In some embodiments, the representation-level pseudo-label generation module and the prompt-level pseudo-label generation module employ a nonlinear support vector machine with the same structure.

[0045] In step S105, this step achieves cross-view semantic alignment and distribution balance simultaneously at the feature layer and label layer through contrastive learning bidirectional loss constraints, completely resolving the clustering performance degradation problem caused by view distribution offset or training instability in traditional methods. Multi-level alignment is performed, and contrastive loss forces the feature layer and pseudo-label layer to cross views. Figure One Consistency is achieved. Forward loss strengthens decision confidence, KL divergence constraints minimize the distribution difference between global cluster labels and fused pseudo-labels, and information entropy penalties are added to suppress low-confidence predictions. Backward feedback stabilization is implemented, and smooth intermediate labels are generated through a transition mapping module to construct backward loss constraints. Finally, a joint loss is constructed based on contrastive loss, forward alignment loss, and backward feedback loss for end-to-end training.

[0046] In some embodiments, the embedding layer structure and the label layer are aligned by contrast learning to establish a contrast loss, and the calculation formula is: ; wherein, represents the contrast loss, represents the local high-level semantic representation corresponding to the kth modal data of the ith sample, represents the modal category number, and m and n represent sample numbers; represents the label matrix of each sample in the mth classification corresponding to the kth modal, m and n represent the category number, N represents the number of samples, and C represents the number of categories; represents a cosine similarity function.

[0047] In some embodiments, the maximum value in the local pseudo label, the fusion pseudo label and the global pseudo label is selected for each classification category to obtain a global clustering label by normalization, including steps S201-S203: Step S201: For the ith sample, the maximum value of the label is selected in the local pseudo label, the fusion pseudo label and the global pseudo label with respect to classification m, and the expression is: ; wherein, represents the global pseudo label of the ith sample with respect to classification m, represents the local pseudo label of the vth modal with respect to classification m in the ith sample, represents the fusion pseudo label of the ith sample with respect to classification m.

[0048] Step S202: The global clustering label of a single sample is calculated, and the calculation formula is: ; wherein, represents the value corresponding to classification p in the global clustering label, represents the maximum value of the label with respect to classification p, represents the maximum value of the label with respect to classification q, and C is the number of classifications.

[0049] Step S203: The calculation formula of the forward alignment loss is: ; wherein, represents the forward alignment loss, represents the local pseudo label matrix of each sample with respect to modal v, represents the global pseudo label matrix, represents the global clustering label matrix, represents the fusion pseudo label matrix; represents the KL divergence; represents information entropy.

[0050] In some embodiments, the local pseudo-label and the global pseudo-label are transition mapped by using a transition mapping module, and the expression is: ; ; wherein, represents a ReLU activation function, represents an intermediate representation learned from the global pseudo-label matrix , represents an intermediate representation learned from the fusion pseudo-label matrix ; , , and are weight matrices; , , and are fixed values.

[0051] The calculation formula of the reverse feedback loss is: ; wherein, represents the reverse feedback loss; The calculation formula of the joint loss is: ; wherein, is the joint loss, and are weights.

[0052] In another aspect, the present application also provides a multi-view clustering method device based on hierarchical prompt guidance alignment, comprising a processor, a memory and computer programs / instructions stored on the memory, wherein the processor is used to execute the computer programs / instructions, and when the computer programs / instructions are executed, the device realizes the steps of the above method.

[0053] In another aspect, the present application also provides a computer readable storage medium, which stores computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the above method are realized.

[0054] The present application will be described below in combination with a specific embodiment: This embodiment provides a scheme for achieving fine-grained semantic alignment and consistent clustering in multi-view scenarios—Hierarchical Prompt-Guided Alignment for Multi-View Clustering (HiPMVC). First, a learnable prompting mechanism is introduced to enhance the original input data at the primary level, preserving and mining the unique semantic information of each view, thereby improving the expressive power of basic feature extraction. Second, at the embedding level, local prompts and global prompts are designed respectively. The former focuses on fine-grained clustering cues within a view, while the latter captures shared semantics between different views, thus compensating for the shortcomings of traditional coarse-grained fusion strategies in capturing complementary information between views. The features obtained after the input is processed by the encoder are further enhanced by fusing modality-specific embedded prompts and global prompts. Subsequently, these processed features and the global prompt are jointly input into the mapping encoder and mapped into a unified feature space. Finally, based on the multi-level labels generated by the pseudo-label classifier, a two-way cue-guided alignment operation is performed. The most representative cluster pseudo-labels are highlighted through forward alignment, and the global and local distributions are adaptively balanced by a feedback mechanism, thereby achieving collaborative alignment at the feature level and the pseudo-label level.

[0055] This embodiment introduces three types of cues: primary level, local embedding level, and global embedding level, to fully extract semantic information from the input data. The input data and global cues are mapped together into a unified latent space by a mapping encoder. Subsequently, high-level semantic features generate pseudo-labels through pseudo-label classifiers at the data level and cue level, and are aligned using a bidirectional cue-guided alignment mechanism, thereby improving multi-level semantic consistency and clustering performance.

[0056] Reference Figure 2 The specific implementation method of this embodiment is as follows: 1. Primary level prompts for learning methods To effectively extract semantic features from the input data, a learnable prompt is introduced for each sample in each modality. Let the multimodal input data be... ,in Indicates the number of samples. Indicates the number of modes. For the first The data dimension has multiple modalities. A multilayer perceptron (MLP) network is designed as the cue encoder. to extract the main-level prompt vector in each modality, the calculation formula is as follows: ; wherein, maintain the same sample quantity and feature dimension as the original input . The prompt vector represents a prompt operation based on the original data, which is used to guide the extraction of semantic information. Subsequently, the input feature is added to its corresponding main-level prompt vector , and the feature embedding network is input to obtain the preliminary semantic feature representation , which is calculated as follows: ; wherein, represents the preliminary semantic feature representation after fusion of the prompt information. To verify the universality of the prompt mechanism, an MLP encoder and a graph convolutional encoder (GCN) are respectively used in the design of .

[0057] 2. Embedding-level prompt learning method In most existing methods, feature processing can generally be divided into single-modal processing and multi-modal fusion across modalities. However, existing fusion mechanisms (such as attention mechanisms) often have difficulty in providing sufficient fine-grained guidance for the extraction of shared semantics, and the fundamental reason lies in the limitation of model capacity and the inherent modeling difficulty in unsupervised tasks. To make up for the above shortcomings, a modality-specific local prompt generation network and a global prompt generation network are introduced at the embedding level. Among them, is used to capture key clustering information in each modality, and is used to model the shared clustering semantics across samples and enhance the global consistency between different instances.

[0058] Similar to the main-level prompt learning mechanism, a lightweight local prompt generation network and a global prompt generation network are designed to generate modality-specific local prompts and global prompts . Among them, the extraction method of the local prompt is as follows: ; wherein, , represents the preliminary semantic feature representation in the th modality. In order to generate the global prompt , cross-modal fusion needs to be performed on the features of all modalities. For this purpose, a set of learnable adaptive weight parameters ​, to dynamically adjust the contribution of each modality feature to the fusion result. The fusion calculation formula is as follows: ; wherein the weight controls the importance of the modality in the fusion process. After obtaining the local prompt and the global prompt , the preliminary semantic feature representation is fused with the prompt information to guide its projection into a higher-level semantic clustering space. The process is defined as follows: ; wherein is a hyperparameter for adjusting the weights of the local and global prompts, represents the local high-level semantic representation after the final projection of the local high-level semantic representation corresponding to the vth modality data of the sample.

[0059] In addition, inspired by the DealMVC method, an attention mechanism-based fusion strategy is introduced to aggregate the feature representations under multiple modalities to generate unified fusion features . The fusion process is defined as follows: The weight coefficients are calculated based on the attention mechanism network and the nonlinear feedforward neural network, and the calculation formula is as follows: ; The local high-level semantic representation corresponding to the vth modality data in the sample is fused based on the weight coefficients, and the calculation formula is as follows: ; wherein represents the Hadamard product (element-wise multiplication), represents the weight matrix of the vth modality data in the sample, represents the attention mechanism network, represents the nonlinear feedforward neural network.

[0060] Since the global prompt itself contains cross-sample shared clustering semantic information, further use of the global prompt, with the help of a projection module with the same structure as , maps it to a higher semantic level clustering space. The process can be represented as: ; 3. Bidirectional prompt-guided alignment mechanism Given the distribution difference between the feature level and the clustering result level of multi-modal data, cross-modal alignment needs to be carried out at both levels. To this end, a double-level alignment strategy based on pseudo-label classifier is proposed to effectively unify the feature representations of different modalities.

[0061] For each high-level feature representation and its mapping result in the unified semantic clustering space , the pseudo-label generation module at the representation level and the pseudo-label generation module at the hint level are defined using the nonlinear multilayer perceptron (MLP) structure , and the following pseudo-label representation is obtained: ; wherein represents the local pseudo-label matrix of each sample with respect to modality v, represents the fused pseudo-label matrix.

[0062] Meanwhile, for the global hint representation after semantic mapping , the global pseudo-label matrix is defined as: ; wherein and use the same nonlinear MLP architecture to output the pseudo-label set .

[0063] After obtaining the representations at the feature level and the pseudo-label level, a contrastive learning strategy is used to align at both levels. Specifically, the constructed contrastive loss function is defined as follows: ; wherein represents the contrastive loss, represents the local high-level semantic representation corresponding to the kth modality data of the ith sample, represents the modality category number, and m and n represent the sample number; represents the label matrix of each sample in the mth classification corresponding to the kth modality, and m and n represent the category number; N represents the number of samples, and C represents the number of categories; represents the cosine similarity function. Wherein, is used to identify the hidden data dimension.

[0064] The first term is used to align the feature level representation, and the second term is used to align the semantic information at the pseudo-label level, thereby realizing multi-modal semantic alignment and consistency modeling. To enhance the consistency of the clustering results and highlight the most representative category labels, a new feedback mechanism is introduced, which includes two stages of forward alignment and backward feedback. ​

[0065] The forward alignment focuses on highlighting the valid clustering results and suppressing irrelevant or wrong class labels. Specifically, the maximum value of all generated labels is taken to obtain the final pseudo-labels: ; wherein, denotes the global pseudo-label of the ith sample with respect to the classification m, denotes the local pseudo-label of the vth modality in the ith sample with respect to the classification m, denotes the fusion pseudo-label of the ith sample with respect to the classification m.

[0066] To further strengthen the representative labels and weaken the influence of irrelevant labels, the global clustering labels are constructed based on the above results, and the expression of the pth class is as follows: ; wherein, denotes the value corresponding to the pth classification in the global clustering label, denotes the maximum value of the label with respect to the pth classification, denotes the maximum value of the label with respect to the qth classification, and C is the number of classifications.

[0067] Finally, the calculation formula of the forward alignment loss is: ; wherein, denotes the forward alignment loss, denotes the local pseudo-label matrix of each sample with respect to the vth modality, denotes the global pseudo-label matrix, denotes the global clustering label matrix, denotes the fusion pseudo-label matrix; denotes the KL divergence; denotes the information entropy. For example, the entropy of is calculated as .

[0068] To realize the backward feedback, a transition mapping module is designed to establish a balance between the global pseudo-label matrix and the fusion pseudo-label matrix , alleviate the instability caused by the direct alignment of the two, and construct a relatively stable clustering representation in the intermediate representation space. The calculation method of the transition mapping is as follows: ; ; wherein, denotes the ReLU activation function, denotes the global pseudo-label matrix intermediate representation learned from the fusion pseudo-label matrix intermediate representation learned from the fusion pseudo-label matrix intermediate representation learned from the fusion pseudo-label matrix , , and are weight matrices. , , and are fixed values. Although in the same class probability space, the distribution shape can be flexibly adjusted after parameterization. Especially in the early stage of model training, this transition mapping module plays a buffering role for direct alignment, which helps the stability of training.

[0069] The calculation formula of the reverse feedback loss is defined as: ;The calculation formula of the joint loss is calculated as: ; wherein, is the joint loss, and are weights. By jointly optimizing this objective function, all network parameters can be automatically learned in an end-to-end manner, and the final output clustering result can be directly generated by the global label .

[0070] This embodiment can be applied in the intelligent manufacturing scene. Multiple heterogeneous sensors are often deployed on the production line: a three-axis vibration probe near the high-speed bearing captures tiny mechanical impacts at a sampling rate of 1 kHz; a thermocouple and an infrared thermometer update temperature changes every second; a current transformer reports the main motor load in real time; an acoustic microphone array listens to abnormal howling at 48 kHz; at the same time, the PLC and SCADA logs of the numerical control system asynchronously write key process parameters and alarm markers. The above vibration, temperature, current, acoustic and control logs can be regarded as five complementary perspectives, which continuously flow to the production line edge gateway at different frequencies from milliseconds to seconds.

[0071] To identify the evolving clusters of device health states and predict potential failures under the premise of "unsupervised", the factory embeds a "hierarchical cue-guided alignment multi-view streaming clustering method" in the edge-cloud collaborative system. The cloud first collects historical multi-view data for months, aligns the timestamps to the finest 10 ms, and unifies the amplitude, temperature, and current magnitude. Then, according to the "double-layer cue" mechanism proposed in the invention: in the main layer, a learnable main cue vector is injected for each view, emphasizing the characteristics of the respective signals; in the embedding layer, local cues and cross-view global cues are generated synchronously, and the five-view features are fused with the help of the attention mechanism. After the model training converges, the system obtains the pseudo-labels, fine-tunes them with k-means, solidifies them into a weight file, and periodically distributes them to the production line edge servers.

[0072] Further, to evaluate the performance of the method, experiments were conducted on seven multi-view datasets: BBCSport, Nuswide, Citeseer, Wiki, Caltech7, and Caltech-all. The brief information of these datasets is summarized in Table 1. The proposed method was compared with nine multi-view clustering baseline methods, including COMIC, CoMVC, SiMVC, MFLVC, MVD, DealMVC, GCFAgg, ICMVC, and VITAL.

[0073] In terms of implementation details, the method described in this embodiment, HiPMVC, is implemented based on PyTorch on a Linux platform and trained on an NVIDIA RTX A6000 GPU. The training is conducted for a total of 500 epochs, and the learning rate is fixed at 0.001. The temperature parameter is set to , . The hyperparameters β and γ are optimized through grid search on {0.01, 0.1, 1, 10, 100}. The results in the parameter analysis section show that the model is not sensitive to these parameters, so finally β = 1.0 and γ = 1.0 are taken.

[0074] Table 1 Seven different multi-view datasets The experimental results are shown in Table 2, which compares the performance difference between the method described in the embodiment and the existing multi-view clustering (MVC) method. The clustering quality is evaluated on seven commonly used data sets, and the evaluation indexes adopted include accuracy (ACC), normalized mutual information (NMI) and purity. The results show that the method adopted in the embodiment achieves better performance on all seven data sets. For example, on the BBCSport data set, the model significantly leads other methods, being 6-7% higher than the second place. In addition, for large-scale data sets such as YoutubeFace, the multi-level prompt design enables the model to better capture the clustering association. Further observation finds that the GCN-based variant is always superior to its corresponding version based on MLP.

[0075] Table 2 shows the clustering performance of HiPMVC on each data set and the comparison with other models as follows: As shown in Figure 3 and 4 , the values of the hyperparameters β and γ set in the joint loss function of the method are in the range of [0.01, 0.1, 1, 10, 100], and experiments are conducted on two data sets of Citeseer and Caltech7. For the Caltech7 data set, the parameter γ shows high sensitivity in the range of 1 to 100; in contrast, for the Citeseer data set with fewer views, the sensitivity of β and γ is relatively stable.

[0076] To visually display the clustering effect of HiPMVC, the t-SNE algorithm is used for visualization. As shown in Figures 5 to 8 , the clustering performance of the model proposed in the embodiment is shown. Compared with the original features ( Figure 5 and Figure 7 ), it can be found that without multi-view fusion, the distribution of each category is chaotic, and there is obvious overlap or loose distribution between some categories. In contrast, after applying our method ( Figure 6 and Figure 8 ), the distribution of sample points in the low-dimensional space becomes more compact and clear, forming well-structured and separated clustering clusters.

[0077] Corresponding to the above method, the application also provides a device / system, which comprises a computer device including a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device / system implements the steps of the method as described above.

[0078] The application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the edge computing server deployment method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0079] In summary, the multi-view clustering method and device based on hierarchical prompt guided alignment can construct a main layer structure and an embedded layer structure to extract semantic features of multi-modal data, and optimize clustering results through a label layer and a bidirectional alignment mechanism. The main layer prompt encoder generates a main layer level prompt vector for each modality and adds it to the original data, and the preliminary semantic feature representation is obtained through a shallow encoder. The original input data can be enhanced, and the unique semantic information of each view can be retained and mined. The local prompt generation network and the global prompt generation network in the embedded layer structure output local prompts and global prompts, respectively. After fusion, the mapping encoder is used for projection to the semantic clustering space to obtain local and global high-level semantic representations. The attention mechanism is used to fuse the local high-level semantic representations of each modality to realize fine-grained semantic enhancement and cross-view semantic alignment optimization. The label layer generates local pseudo labels, fused pseudo labels, and global pseudo labels of each modality, and obtains global clustering labels by selecting the maximum value and normalizing. The most discriminative signal is strengthened, and the interference of low confidence views is weakened. The contrast loss of the embedded layer structure and the label layer is established through contrast learning, the forward alignment loss is constructed by combining the distribution probability difference between the global clustering labels and the fused pseudo labels, and the reverse feedback loss is constructed by using the transition mapping module. Finally, the parameters are updated based on the joint loss to generate the clustering results, which takes into account the fine-grained semantic mining within the view and the global consistency between the views, and improves the clustering accuracy and stability.

[0080] Those of ordinary skill in the art will appreciate that the various illustrative components, systems and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, or both. The particular implementation is dependent on the specific application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. When implemented in hardware, for example, the hardware can comprise an electronic circuit, an Application Specific Integrated Circuit (ASIC), a suitable firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the application are the program or code segments to perform a specific task. The program or code segments can be stored in a machine-readable medium, or transmitted by a carrier wave as data signals over a transmission medium or communication link.

[0081] It is to be understood that the application is not limited to the particular configurations and processes described herein and shown in the drawings, which can be varied in accordance with the particular needs of the application. For the sake of brevity, conventional techniques and methods related to making and using the application can not be described in detail herein. In the above embodiments, several specific steps are described and illustrated in order to provide a thorough understanding of the present application. However, the process of the present application can be practiced with less than all of the described specific steps, or with additional steps, or with different order of the steps, without departing from the spirit of the application.

[0082] In the present application, features described and / or illustrated in connection with one embodiment can be used in the same or a similar way or in conjunction with or in place of features of another embodiment.

[0083] The above description is intended to be illustrative and not restrictive. Many other changes and modifications can occur to those skilled in the art once advised of the principles of the application. Any such changes and modifications including without limitation those relating to the methods of operation and the specific designs of the systems and methods are intended to be included within the scope of the application. Therefore, the scope of the application is not to be determined solely by the preferred embodiments, and the intent and proper scope of the application is limited only by the appended claims.

Claims

1. A multi-view clustering method based on hierarchical prompts and alignment, characterized in that, The method includes the following steps: For a target scenario, obtain multi-view data to be clustered. The multi-view data contains multiple samples, and each sample has multiple modal views. A main layer structure is constructed. A main layer cue encoder is used to take the sample as input and output the main layer cue vector of each modality. The main layer cue vector of each modality is added to the original data and input into the shallow layer encoder to obtain the preliminary semantic feature representation of each modality. An embedding layer structure is constructed. For each modality, a local prompt generation network is established, taking the preliminary semantic feature representation of the corresponding modality as input and outputting local prompts. A global prompt generation network is established, taking the preliminary semantic feature representations fused based on adaptive weight parameters as input and outputting global prompts. The preliminary semantic feature representation, local prompts, and global prompts corresponding to each modality in the sample are fused and projected onto the semantic clustering space through a first mapping encoder to obtain the corresponding local high-level semantic representation. The global prompts are projected onto the semantic clustering space through a second mapping encoder to obtain the corresponding global high-level semantic representation. An attention mechanism is introduced to fuse the local high-level semantic representations of each modality in the sample to obtain a fused semantic representation. A label layer is constructed by inputting the local high-level semantic representation of each modality of the sample into the representation-level pseudo-label generation module to obtain the local pseudo-labels corresponding to each modality, and inputting the fused semantic representation into the representation-level pseudo-label generation module to obtain the fused pseudo-labels; the global high-level semantic representation is input into the prompt-level pseudo-label generation module to obtain the global pseudo-labels; for each classification category, the maximum value among the local pseudo-labels, the fused pseudo-labels, and the global pseudo-labels is selected and normalized to obtain the global clustering label; A contrastive learning method is used to align the embedding layer structure and the label layer to establish a contrastive loss. The probability distribution difference between the global clustering label and the fused pseudo-label is calculated, and the information entropy of the local pseudo-label and the global pseudo-label is suppressed to construct a forward alignment loss. A transition mapping module is used to perform transition mapping on the local pseudo-label and the global pseudo-label, and the probability distribution difference between them and the original global pseudo-label and global clustering label is calculated to construct a back feedback loss. Based on the contrastive loss, the forward alignment loss, and the back feedback loss, a joint loss is constructed to update the parameters of the embedding layer structure, the label layer, and the transition mapping module. Clustering results are generated based on the updated global clustering label.

2. The multi-view clustering method based on hierarchical prompt guidance alignment according to claim 1, characterized in that, The main layer prompt encoder is a multilayer perceptron encoder, and the shallow layer encoder is a multilayer perceptron encoder or a graph convolution encoder; the representation-level pseudo-label generation module and the prompt-level pseudo-label generation module are nonlinear support vector machines with the same structure.

3. The multi-view clustering method based on hierarchical prompt guidance alignment according to claim 2, characterized in that, A global prompt generation network is established, taking the preliminary semantic features fused based on adaptive weight parameters as input and outputting global prompts. The calculation formula is as follows: ； in, This indicates the global suggestion. This refers to the global suggestion generation network. This represents the adaptive weight parameter for the v-th modality data of the sample. The preliminary semantic feature representation of the v-th modality data of the sample.

4. The multi-view clustering method based on hierarchical prompting and alignment according to claim 3, characterized in that, The preliminary semantic feature representation, the local cue, and the global cue corresponding to each modality in the sample are fused and projected onto the semantic clustering space through the first mapping encoder to obtain the corresponding local high-level semantic representation. The calculation formula is as follows: ； in, This represents the local high-level semantic representation corresponding to the v-th modality data of the sample. This refers to the first mapping encoder. This represents the v-th modal data of the sample. These are preset parameters. This indicates the local cue corresponding to the vth modal data of the sample.

5. The multi-view clustering method based on hierarchical prompt guidance alignment according to claim 4, characterized in that, The local high-level semantic representations of each modality of the sample are fused using an attention mechanism to obtain a fused semantic representation, including: The weight coefficients are calculated based on attention mechanism networks and nonlinear feedforward neural networks, using the following formula: ； Based on the weighting coefficients, the local high-level semantic representations corresponding to v modal data in the sample are fused, and the calculation formula is as follows: ； in, This represents element-wise multiplication. This represents the weight matrix of the v-th modality data in the sample. This represents the attention mechanism network. This represents the nonlinear feedforward neural network. This represents the fused semantic representation.

6. The multi-view clustering method based on hierarchical prompt guidance alignment according to claim 1, characterized in that, Contrastive learning is used to align the embedding layer structure and the label layer to establish a contrastive loss, calculated as follows: ； in, This represents the contrast loss. The local high-level semantic representation corresponding to the k-th modality data of the i-th sample is represented by the following. The modality category number is represented by m and n, which represent the sample number. This represents the label matrix for each sample in the k-th modality, where m and n represent the class numbers; N represents the number of samples, and C represents the number of classes. This represents the cosine similarity function. For the temperature parameters of the embedded structural layer, This refers to the temperature parameters for the label layer.

7. The multi-view clustering method based on hierarchical prompt guidance alignment according to claim 6, characterized in that, For each classification category, the maximum value among the local pseudo-label, the fused pseudo-label, and the global pseudo-label is selected and normalized to obtain the global clustering label, including: For the i-th sample with respect to classification m, the maximum value of the label is selected from the local pseudo-label, the fused pseudo-label, and the global pseudo-label, expressed as: ； in, Let represent the global pseudo-label of the i-th sample with respect to classification m. Let represent the local pseudo-label of the v-th modality in the i-th sample with respect to classification m. The fused pseudo-label of the i-th sample with respect to classification m; The global cluster label for a single sample is calculated using the following formula: ； in, This represents the value corresponding to category p in the global clustering label. This represents the maximum value of the label with respect to category p. Let C represent the maximum value of the label for category q, where C is the number of categories; The formula for calculating the forward alignment loss is: ； in, This represents the forward alignment loss. This represents the local pseudo-label matrix of each sample with respect to mode v. Represents the global pseudo-label matrix. Represents the global clustering label matrix. This represents the fused pseudo-label matrix; Indicates KL divergence; This represents information entropy.

8. The multi-view clustering method based on hierarchical prompt guidance alignment according to claim 7, characterized in that, The transition mapping module is used to perform transition mapping on the local pseudo-tags and the global pseudo-tags, and the expression is: ； ； in, Represents the ReLU activation function. This indicates the origin from the global pseudo-label matrix. The intermediate representation obtained from learning. This indicates the fusion pseudo-label matrix The intermediate representation obtained from learning; , , and This is the weight matrix; , , and It is a fixed value; The formula for calculating the reverse feedback loss is: ； in, This represents the reverse feedback loss; The formula for calculating the joint loss is: ； in, For the aforementioned joint loss, and For weights.

9. A multi-view clustering method apparatus based on hierarchical prompting and alignment, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 8.

Images