Pseudo-label guided semantic clustering method for unannotated multi-modal dialogue

Abstract

The application relates to a pseudo-label guided semantic clustering method for unlabeled multi-modal dialogue and belongs to the field of natural language processing. In the multi-modal representation learning stage, offline semantic features are extracted through a frozen large multi-modal language model, and the representation of a trainable encoder is anchored to high-level intent semantics through a cosine alignment loss; in the semantic division stage, a fixed text neighbor table is constructed using a frozen text encoder, and cross-sample neighbor positive examples are used to model the semantic neighborhood structure between samples during pre-training; in the pseudo-label iterative refinement stage, a density-interval joint scoring criterion is used to select high-confidence cluster center samples, supervised contrast learning is applied, and unsupervised consistency learning is performed on low-confidence boundary samples. The method provides stable semantic anchors for cross-sample supervision, effectively makes up for the deficiency that instance-level contrast targets cannot perceive semantic relationships between samples, and greatly improves the training stability and result reproducibility of unsupervised multi-modal semantic discovery.

Description

Technical Field

[0001] This invention relates to a pseudo-label-guided semantic clustering method for unlabeled multimodal dialogues, belonging to the field of natural language processing. Background Technology

[0002] Unlabeled multimodal clustering aims to uncover the latent semantic structure in dialogue data by jointly modeling heterogeneous modal signals such as text, audio, and video, without relying on manual annotation. With the continuous growth of large-scale dialogue corpora, their semantic organization is becoming increasingly important for intent understanding, interaction analysis, and downstream discourse modeling. Grouping discourses with similar communicative meanings can simultaneously support corpus-level exploration and practical semantic modeling, while reducing reliance on expensive annotation resources.

[0003] While existing unsupervised intent discovery methods each have their own focus, they share a common three-stage paradigm: the first stage is multimodal representation learning, mapping the original multimodal inputs to a shared embedding space; the second stage is semantic partitioning, dividing the embedding space into several latent semantic groups using clustering algorithms; and the third stage is pseudo-label-guided iterative refinement, using the clustering assignment results as pseudo-labels to iteratively refine the representation space through a "clustering-update-re-clustering" loop. Representative methods include Deep Embedding Clustering (DEC), Deep Clustering, Invariant Information Clustering (IIC), and multimodal intent discovery systems such as MCN and UMC.

[0004] The aforementioned methods suffer from three systemic flaws. First, there is a gap in semantic representation: existing methods rely on modality-specific encoders primarily optimized for low-level feature extraction or instance-level alignment, resulting in insufficient stimulation for cross-modal semantic calibration and high-level abstraction. This leads to the learned embedding space reflecting surface modal patterns rather than underlying semantic content; when modal inconsistencies or noise exist, semantically similar utterances may be separated due to differences in acoustic or visual cues. Second, there is blindness to cross-sample relationships: existing methods generally employ instance-level contrastive targets in the multimodal representation learning stage, aligning only the enhanced views of the same sample and completely ignoring semantic relationships between samples. This prevents utterances from the same latent category from forming a compact cohesive structure in the embedding space. Third, there is iterative refinement without confidence perception: existing methods directly reuse cluster assignments as pseudo-labels in the iterative refinement stage, implicitly treating all samples as equally reliable. However, in practice, cluster center samples are far more reliable than boundary samples. Indiscriminate reuse of pseudo-labels leads to the continuous propagation and amplification of early misassignments in multiple iterations. Summary of the Invention

[0005] The purpose of this invention is to address the three shortcomings of existing unlabeled multimodal clustering methods: semantic representation gap, blindness to cross-sample relationships, and lack of confidence-aware refinement. This invention provides a pseudo-label-guided semantic clustering method for unlabeled multimodal dialogues. A unified framework is constructed through three collaborative components: semantic calibration, relationship enhancement, and quality-aware refinement, thereby improving clustering performance and training stability.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] The technical features of this invention are as follows: A shared multimodal encoder is constructed, with BERT as the text branch and a single-layer Transformer encoder as the video and audio branches, fusing the three modal features of each utterance into a unified embedding representation. During the pre-training phase, modality-dropping instance-level contrastive learning, freezing large-scale multimodal language model semantic alignment, and cross-sample relation enhancement learning based on frozen text neighbors are jointly performed. In the iterative refinement phase, high-confidence samples are selected using density-margin joint scoring to apply supervised contrastive learning, while low-confidence samples are only subjected to unsupervised instance-level consistency learning, forming three collaborative modules: semantic calibration, relation enhancement, and quality-aware refinement. These three modules work together, with semantic calibration achieving multimodal semantic space alignment, relation enhancement modeling semantic associations between samples, and quality-aware refinement controlling the reliable propagation of pseudo-label supervision signals.

[0008] This invention provides a semantic discovery method for unlabeled multimodal dialogue data, which is implemented through the following steps:

[0009] S1. Construct a shared multimodal encoder

[0010] The text token sequence, video feature sequence, and audio feature sequence for each utterance are encoded using three independent branches. Specifically:

[0011] The S1.1 text branch uses a pre-trained BERT encoder to process the text token sequence, and after taking the CLS token representation, it is mapped to the fusion dimension through a linear projection layer.

[0012] The S1.2 video branch applies a modality-specific linear projection to the video feature sequence and then processes it through a single-layer Transformer encoder, taking the final time step output as the video sequence embedding.

[0013] The audio branch structure in S1.3 is the same as the video branch. The same operation is performed on the audio feature sequence to obtain the audio sequence embedding.

[0014] S1.4 concatenates the embedding vectors from S1.1, S1.2, and S1.3, then sequentially passes them through a GELU activation function, random dropout, and a linear fusion layer to obtain a fused representation. This fused representation is then mapped to the contrast space using a lightweight linear projector. All loss calculations use the L2-normalized vector output by the projector. During the training phase, the text branches are fine-tuned end-to-end. During the semantic segmentation and inference phases, K-means clustering is performed using the L2-normalized fused representation output by the fusion layer, where K represents the preset number of cluster categories.

[0015] S2, Semantic calibration pre-training

[0016] Based on the shared multimodal encoder output obtained in step S1, semantic calibration pre-training is performed. During the multimodal representation learning stage, the encoder is jointly trained using modality dropout contrastive learning and large-scale multimodal language model semantic alignment loss to anchor the embedding space to high-level intent semantics. Specifically:

[0017] S2.1 Constructs three types of modal discard views for each utterance: the first type of view retains text and audio and discards video; the second type of view retains text and video and discards audio; the third type of view retains all modalities of text, video and audio.

[0018] Text modalities are preserved across all three view types, providing the most stable semantic signals. For the same utterance, each of the three view types is paired as a positive example, and a supervised contrastive loss function is used to calculate instance-level cross-view alignment loss. As shown in equation (1), this encourages modality-invariant representations.

[0019] (1)

[0020] Where B represents the set of samples in a training mini-batch. This represents the L2-normalized projection vector of the i-th sample in the full modal view. This represents the average representation of the normalized projection vector corresponding to the i-th sample under the discarded views of the other modalities.

[0021] S2.2 introduces a semantic alignment mechanism for large-scale multimodal language models.

[0022] Offline extraction of several hidden states from the upper layers of a large-scale multimodal language model for each utterance is performed. Tokens from text, audio, and video routes are average-pooled separately. The aggregated features from each layer are weighted and summed according to preset weights, then L2 normalized and concatenated to form a teacher semantic representation. This teacher semantic representation remains fixed throughout the training process. Since the teacher embedding dimension differs from the encoder fusion dimension, a learnable linear projection layer maps the teacher representation to the encoder fusion dimension. Minimizing this loss drives the encoder representation to converge towards the intent-aware semantics inherent in the large-scale multimodal language model.

[0023] S3, Relationship Enhancement Pre-training

[0024] After completing the semantic calibration pre-training in step S2, the relationship enhancement pre-training stage begins. Using the shared multimodal encoder constructed in step S1 as the training object, a fixed text nearest neighbor table is constructed by freezing the text encoder during the fusion representation learning process involving text, video, and audio modalities. Relationship enhancement pre-training is then carried out using cross-sample nearest neighbor positive examples, and the parameters of the shared multimodal encoder are updated with the total pre-training loss.

[0025] Before the pre-training loop begins, the CLS representation of the frozen text encoder is extracted from all training samples and L2 normalized. The cosine similarity matrix of all samples is calculated, and the indices of the highest cosine similarity samples are retained for each sample to form a fixed text nearest neighbor table. This fixed text nearest neighbor table remains unchanged throughout the entire pre-training process.

[0026] In each training mini-batch, a binary positive pair mask is constructed based on the global sample index and a fixed nearest neighbor list. The nearest neighbor relationships are bidirectionally unioned to ensure symmetry. Supervised contrastive loss is used to calculate the intra-batch cross-sample nearest neighbor alignment loss. Only samples with at least one positive nearest neighbor within the batch are retained for loss calculation, encouraging semantically related discourses to form cohesive local neighborhood structures in the embedding space. The probability of a sample in a mini-batch having at least one positive nearest neighbor is approximated using a hypergeometric distribution, serving as a reference for selecting the size of the nearest neighbor list.

[0027] The total pre-training loss is calculated as shown in equation (2):

[0028] (2)

[0029] in, Indicates the total pre-training loss, Indicates instance-level modality drop contrast loss, Indicates the nearest neighbor comparison loss across sample texts, This represents the semantic alignment loss of a large multimodal language model. This represents the weights of the nearest neighbor comparison loss across sample texts. This represents the semantic alignment loss weights for large-scale multimodal language models.

[0030] S4. Pseudo-label generation and quality-perceived sample screening

[0031] After completing the pre-training for relationship enhancement in S3, the pseudo-label iterative refinement stage begins. High-confidence samples are selected from the K-means clustering results of each round using the density-margin joint scoring criterion, and supervised contrastive learning is applied to them. Unsupervised consistency learning is applied to the low-confidence samples.

[0032] After pre-training, the iterative refinement phase begins. At the start of each refinement iteration, all training samples are forward-propagated through the encoder to extract fused representations and L2 normalized. K-means clustering is then performed on the normalized features, where K represents the preset number of clusters, and pseudo-labels are assigned to each sample. The initial clustering uses K-means with a plus sign for initialization, and multiple random restarts are performed to avoid initialization bias. Subsequent rounds use the cluster centroids from the previous round as warm-start initializations, and only a single initialization is performed to reduce computational overhead. K-means clustering is performed on the L2 normalized features using Euclidean distance, and the cluster centroids are also L2 normalized before distance evaluation to ensure consistency in the distance metrics used for clustering and sample selection.

[0033] Density-margin joint scoring is applied to the clustering results, and high-confidence samples are selected based on the comprehensive reliability score of each sample. The training objective for the high-confidence samples consists of two parts: the first part is a supervised contrastive loss using K-means pseudo-labels as class supervision signals, where other selected samples from different modal dropout views of the same utterance and from the same pseudo-label class are considered positive examples; the second part is a cosine alignment loss with the frozen teacher representation of the large multimodal language model. The encoder is updated in an independent gradient step after a weighted sum of the two losses. The training objective for the low-confidence samples is only instance-level cross-views between different modal dropout views of the same utterance. Figure 1 Consistent loss is used, and no pseudo-label supervision is employed. Gradient updates for both classes of samples are performed independently within the same round, without interference.

[0034] The calculation steps of the density-interval joint scoring criterion are as follows: calculate the density score and interval score for each sample within each cluster; the density score is calculated as follows: for each cluster, adaptively determine the query size, take the larger value between a certain proportion of the cluster size and a fixed lower bound, estimate the average distance of each sample to its nearest neighbor in the same cluster using K-nearest neighbor reachability, take the inverse of the distance as the original density reliability, and normalize it within the cluster to obtain the density score; the interval score is calculated as follows: calculate the difference between the distance of each sample to the centroid of the assigned cluster and the distance to the centroid of the nearest competing cluster as the original interval, and normalize it within the cluster to obtain the interval score. The comprehensive reliability score is a weighted average of the density score and the interval score, as shown in Equation (3):

[0035] (3)

[0036] in, This represents the overall reliability score of the i-th sample. This represents the density score of the i-th sample. This represents the interval score of the i-th sample. This represents the balance coefficient between the density score and the interval score.

[0037] The samples within each cluster are sorted from highest to lowest based on their overall scores. A certain percentage of these samples are selected as the high-confidence selected set, while the rest are selected as the low-confidence unselected set. The selection percentage increases incrementally from its initial value with each training epoch until a preset upper limit is reached, gradually expanding the size of the supervised subset.

[0038] S5, Quality Perception Comparison Refinement Training

[0039] After completing pseudo-label generation and quality-aware sample screening in step S4, supervised contrastive loss is calculated using pseudo-labels for high-confidence selected samples, and semantic alignment loss is added. The encoder is updated in an independent gradient step. For low-confidence unselected samples, the encoder is updated only using instance-level consistency loss among the three modalities. Steps S4 and S5 are executed iteratively until a preset number of refinement rounds are reached, and the final clustering result is output. During inference, the samples to be processed are input into the trained shared multimodal encoder, its fused representation is extracted, and clustering labels are assigned according to the nearest centroid rule, thus realizing a pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue data.

[0040] Beneficial effects

[0041] 1. The pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue disclosed in this invention proposes the first method that integrates semantic calibration, cross-sample relationship modeling, and quality-aware pseudo-label refinement into a unified multimodal clustering framework. It systematically solves the three core defects of existing unlabeled multimodal clustering methods and achieves optimal or near-optimal comprehensive performance on the three benchmarks of MIntRec, MELD-DA, and IEMOCAP-DA.

[0042] 2. The pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue disclosed in this invention uses a fixed text nearest neighbor list to construct cross-sample semantic neighborhoods. The nearest neighbor list remains fixed throughout the pre-training process and is not affected by noise from early multimodal clustering, providing stable semantic anchors for cross-sample supervision and effectively making up for the deficiency that instance-level comparison targets cannot perceive the semantic relationships between samples.

[0043] 3. The pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue disclosed in this invention considers two complementary dimensions, intra-cluster compactness and inter-cluster separability, in a density-margin joint scoring criterion. This is superior to screening strategies that use density or margin alone. It effectively suppresses the propagation of pseudo-label noise in samples with ambiguous boundaries and reduces the variance between random seeds by more than 50% on multiple indicators. This significantly improves the training stability and reproducibility of unsupervised multimodal semantic discovery. Attached Figure Description

[0044] Figure 1 This is a flowchart of the semantic discovery method for unlabeled multimodal dialogue data according to the present invention. Detailed Implementation

[0045] To better illustrate the purpose and advantages of the present invention, the invention will be further described below in conjunction with the accompanying drawings and examples.

[0046] Example

[0047] All models in this embodiment are implemented in PyTorch and trained on a single NVIDIA A100-PCIE-40GB GPU. The text branch uses a BERT-BASE encoder, and the multimodal fusion dimension is set to 256. The batch size is 128, the contrast temperature is 0.15, and the number of pre-training epochs and iterative refinement epochs are both configured in (50, 30) on three datasets. The pre-training weights and refinement weights of the semantic alignment loss are set to 0.03 and 0.005, respectively. The density-margin balance coefficient is set to 0.5, and the selection ratio is increased from an initial value of 0.05 in increments of 0.02 per epoch until a maximum of 0.35 is reached. The teacher semantic features were extracted offline from the Qwen2.5-Omni-7B multimodal large-scale language model. The upper layers -4, -8, and -12 were taken, and the aggregated features of each layer were weighted and summed with weights of 0.4, 0.35, and 0.25, respectively. The three-way route features were concatenated to obtain a teacher representation with a dimension of 10752, which was then mapped to a 256-dimensional fusion space through a learnable linear projection layer.

[0048] This embodiment focuses on a semantic discovery method for unlabeled multimodal dialogue data, such as... Figure 1 As shown, this is achieved through the following steps:

[0049] Step 1: Construct a shared multimodal encoder

[0050] This embodiment constructs a three-branch shared multimodal encoder. The text branch processes the text token sequence using a BERT-BASE encoder, extracts the CLS token representation, and then maps it to 256 dimensions through a linear projection layer. The video branch applies a modality-specific linear projection to the original video feature sequence, processes it through a single-layer Transformer encoder, and takes the final time step output as the video embedding. The audio branch has the same structure as the video branch. The three 256-dimensional embedding vectors are concatenated to obtain a 768-dimensional representation, which is then sequentially activated by GELU, randomly discarded, and processed by a 768-dimensional to 256-dimensional linear fusion layer to obtain a 256-dimensional fused representation. Finally, it is mapped to the contrast space through a 256-dimensional to 256-dimensional lightweight linear projection head.

[0051] Step 2: Semantic calibration pre-training

[0052] Construct three modal discard views for each utterance. Figure 1View 1 retains text and audio, replacing video input with an all-zero tensor; View 2 retains text and video, replacing audio input with an all-zero tensor; View 3 retains all modalities as complete modal anchors. The three types of views form pairwise positive pairs, and instance-level cross-view alignment loss is calculated using a supervised contrastive loss function.

[0053] Simultaneously, the hidden states of each utterance in the frozen Qwen2.5-Omni-7B dataset were extracted offline to construct a 10752-dimensional teacher semantic representation. After being mapped to 256 dimensions through a learnable linear projection layer, the instance-level cross-view alignment loss was calculated using a supervised contrastive loss function. As shown in Equation (1), this encourages modality-invariant representations, driving the encoder representations to move closer to intention-aware semantics.

[0054] (4)

[0055] Where B represents the set of samples in a training mini-batch. This represents the L2-normalized projection vector of the i-th sample in the full modal view. This represents the average representation of the normalized projection vector corresponding to the i-th sample under the discarded views of the other modalities.

[0056] Step 3: Relationship Enhancement Pre-training

[0057] Before the pre-training loop begins, the CLS representations of all training samples of the frozen BERT are extracted and L2 normalized. The cosine similarity matrix of all samples is calculated, and fixed text nearest neighbor tables with 15, 30, and 10 nearest neighbors are maintained for the MintRec, MELD-DA, and IEMOCAP-DA datasets, respectively. Subsequently, in the fusion representation learning process involving text, video, and audio modalities, cross-sample positive nearest neighbor pairs are constructed in each training mini-batch based on the fixed text nearest neighbor tables, and the cross-sample nearest neighbor contrastive loss is calculated accordingly. Relational augmentation pre-training is then performed on the shared multimodal encoder.

[0058] In each training mini-batch, a symmetric binary positive pair mask is constructed based on the fixed nearest neighbor list, and the cross-sample nearest neighbor contrast loss is calculated for valid samples that have at least one nearest positive neighbor within the batch. The encoder is updated by weighting and summing the three losses using Equation (5) as the total pre-training loss.

[0059] (5)

[0060] in, Indicates the total pre-training loss, Indicates instance-level modality drop contrast loss, Indicates the nearest neighbor comparison loss across sample texts, This represents the semantic alignment loss of a large multimodal language model. This represents the weights of the nearest neighbor comparison loss across sample texts. This represents the semantic alignment loss weights for large-scale multimodal language models.

[0061] Step 4: Pseudo-label generation and quality-perceived sample screening

[0062] After completing the semantic calibration pre-training in step two and the relation enhancement pre-training in step three, the iterative refinement phase begins, which lasts for 30 rounds. At the beginning of each iteration, the L2 normalized fusion representation of all training samples is extracted, and K-means clustering is performed on the normalized feature space, where K represents the preset number of cluster categories. The first clustering is initialized using K-means++, and 20 random restarts are performed to reduce initialization bias; in subsequent rounds, the cluster centroids obtained from the previous iteration are used as warm-start initializations, and only a single initialization is performed to reduce computational overhead.

[0063] Density-interval joint scoring is performed on samples within each cluster. The adaptive query size is the larger of 20% of the cluster size and 2. The density score is obtained by local density estimation based on the K_nn nearest neighbor distance and normalized within the cluster, where K_nn represents the number of nearest neighbors used for local density estimation and is independent of the number of cluster categories K in K-means clustering. The interval score is calculated based on the difference between the distance of the sample to the centroid of the nearest competing cluster and the distance to the centroid of the assigned cluster, and normalized within the cluster. The density score and the interval score are weighted by a balance coefficient of 0.5 according to Equation (6) to obtain the comprehensive reliability score of the sample. The high-confidence sample set is determined by sorting according to the comprehensive score, and the selection ratio starts from 0.05 and increases by 0.02 in each round until it reaches 0.35.

[0064] (6)

[0065] in, This represents the overall reliability score of the i-th sample. This represents the density score of the i-th sample. This represents the interval score of the i-th sample. This represents the balance coefficient between the density score and the interval score.

[0066] Step 5: Quality Perception Comparison Refinement Training

[0067] After completing the pseudo-label generation and quality-aware sample screening in step four, for high-confidence selected samples, the pseudo-labels generated by clustering are used as supervision signals to calculate supervised contrastive loss, and semantic alignment loss is superimposed to update the parameters of the shared multimodal encoder, where the weight of semantic alignment loss is a preset value; for low-confidence unselected samples, only the instance-level consistency loss between different modal discarded views of the same discourse is used for encoder optimization, without pseudo-label supervision, in order to reduce the impact of erroneous pseudo-label propagation on model training.

[0068] The pseudo-label generation and quality-aware sample screening in step four, as well as the quality-aware comparison and refinement training in this step, are executed iteratively until the preset number of training rounds is reached. This yields the final shared multimodal encoder parameters, cluster centroids, and clustering results for all samples, thus realizing a pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue data.

[0069] On the MintRec dataset, the proposed method outperforms the baseline UMC in four metrics: normalized mutual information, adjusted Rand index, clustering accuracy, and Fowlkes-Mallows index. The average scores for these four metrics reach 37.43, 29.24, and 28.13, respectively, an improvement from 36.76 to 37.43. On the MELD-DA and IEMOCAP-DA datasets, improvements are seen in the three pairwise consistency metrics: adjusted Rand index, clustering accuracy, and Fowlkes-Mallows index. A comparison of the stability of the five-seed adjusted Rand index distribution shows that the proposed method reduces the inter-seed variance of multiple metrics by more than 50% compared to the component-free baseline, validating the effectiveness of the three-component co-design in improving training stability.

[0070] This embodiment evaluates six ablation variants by switching three binary components—semantic alignment of a large multimodal language model, density-margin boundary-sensitive filtering, and cross-sample pre-training of text nearest neighbors. On MIntRec and MELD-DA, the mean score monotonically increases as the three components are progressively introduced along the cumulative path, indicating that the three components are complementary rather than redundant. On IEMOCAP-DA, semantic alignment provides the largest single-step gain; subsequent components primarily improve training stability by reducing variance rather than monotonically increasing the mean, reflecting the characteristic of quality-aware refinement prioritizing reliability over peak mean in highly ambiguous dialogue behavior data. Hyperparameter sensitivity analysis shows that the text nearest neighbor table size is stable within a reasonable range, the text nearest neighbor loss weights are robust over a wide value range, and a density-margin balance coefficient of 0.5 is superior to pure density or pure margin endpoints on MIntRec.

[0071] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue, characterized in that, This is achieved through the following steps: S1. Construct a shared multimodal encoder The text token sequence, video feature sequence, and audio feature sequence of each utterance are encoded through independent branches and then fused into a unified embedding representation; specifically: The S1.1 text branch uses a pre-trained BERT encoder to process the text token sequence, takes the CLS token representation, and then maps it to the fusion dimension through a linear projection layer; S1.2 The video branch applies a modality-specific linear projection to the video feature sequence and then processes it through a single-layer Transformer encoder, taking the final time step output as the video sequence embedding; The audio branch structure in S1.3 is the same as the video branch. The same operation is performed on the audio feature sequence to obtain the audio sequence embedding. S1.4 concatenates the three embedding vectors of S1.1, S1.2, and S1.3, and then sequentially passes them through the GELU activation function, random dropout, and linear fusion layer to obtain the fused representation. It is then mapped to the contrast space by a lightweight linear projection head. All loss calculations use the L2 normalized vector output by the projection head. During the training phase, the text branches are fine-tuned end-to-end; In the semantic segmentation and reasoning stage, K-means clustering is performed using the L2 normalized fusion representation output by the fusion layer, where K represents the preset number of cluster categories; S2, Semantic calibration pre-training Based on the shared multimodal encoder output obtained in step S1, semantic calibration pre-training is performed. In the multimodal representation learning stage, the encoder is jointly trained with modality dropout contrastive learning and large multimodal language model semantic alignment loss to anchor the embedding space to high-level intent semantics. Specifically: S2.1 Constructs three types of modal discard views for each utterance: the first view retains text and audio, and discards video; the second view retains text and video, and discards audio; the third view retains all modalities of text, video, and audio. Text modalities are preserved in all three view types. For the same utterance, each of the three view types is set as a positive pair, and a supervised contrastive loss function is used to calculate the instance-level cross-view alignment loss. As shown in equation (1): (1) Where B represents the set of samples in a training mini-batch. This represents the L2-normalized projection vector of the i-th sample in the full modal view. This represents the average representation of the normalized projection vector of the i-th sample under the discarded views of the other modalities; S2.2 introduces a semantic alignment mechanism for large-scale multimodal language models. Offline extraction of several hidden states at the upper layers of each utterance in a large-scale multimodal language model is performed. The tokens of text routing, audio routing, and video routing are averaged and pooled separately. The aggregated features of each layer are weighted and summed according to preset weights. After L2 normalization, the three features are concatenated to form a teacher semantic representation. This teacher semantic representation remains fixed throughout the training process. The teacher representation is mapped to the encoder fusion dimension through a learnable linear projection layer. Minimizing this loss drives the encoder representation to move closer to the intent-aware semantics implied by the large-scale multimodal language model. S3, Relationship Enhancement Pre-training After completing the semantic calibration pre-training in step S2, the relationship enhancement pre-training stage begins. The shared multimodal encoder from step S1 is used as the training object. In the fusion representation learning process involving text, video, and audio modalities, a fixed text nearest neighbor table is constructed using the frozen text encoder. Relationship enhancement pre-training is carried out using cross-sample nearest neighbor positive examples. The parameters of the shared multimodal encoder are updated with the total pre-training loss. Before the pre-training loop begins, the CLS representation of the frozen text encoder is extracted from all training samples and L2 normalized. The cosine similarity matrix of all samples is calculated. For each sample, the indices of the nearest neighbor samples with the highest cosine similarity are retained to form a fixed text neighbor table. This fixed text neighbor table remains unchanged throughout the pre-training process. In each training mini-batch, a binary positive pair mask is constructed based on the global sample index and the fixed nearest neighbor table. The nearest neighbor relationship is taken as a bidirectional union to make the relationship symmetric. The intra-batch cross-sample nearest neighbor alignment loss is calculated using supervised contrastive loss. Only samples with at least one positive nearest neighbor in the batch are retained to participate in the loss calculation, which encourages semantically related discourses to form a cohesive local neighborhood structure in the embedding space. S4. Pseudo-label generation and quality-perceived sample screening After completing the pre-training for relationship enhancement in S3, the pseudo-label iterative refinement stage begins. High-confidence samples are selected from the K-means clustering results of each round using the density-margin joint scoring criterion, and supervised contrastive learning is applied to them. Unsupervised consistency learning is applied to the low-confidence samples. At the beginning of each refinement iteration, all training samples are forward-propagated through the encoder to extract fused representations and L2 normalized. K-means clustering is then performed on the normalized features, where K represents the preset number of cluster categories, and pseudo-labels are assigned to each sample. The first clustering uses K-means plus sign initialization and performs multiple random restarts to avoid initialization bias. Subsequent rounds use the cluster centroids from the previous round as warm-start initialization and perform only a single initialization to reduce computational overhead. K-means clustering is performed on L2 normalized features using Euclidean distance, and the cluster centroids are also L2 normalized before distance evaluation to ensure consistency of the distance metrics used for clustering and sample selection. The clustering results are subjected to density-margin joint scoring, and high-confidence samples are selected based on the comprehensive reliability score of each sample. The calculation steps of the density-interval joint scoring criterion are as follows: calculate the density score and interval score for each sample within each cluster; the density score is calculated as follows: for each cluster, adaptively determine the query size, take the larger value between a certain proportion of the cluster size and a fixed lower bound, estimate the average distance of each sample to its nearest neighbors in the same cluster using K-nearest neighbor reachability, take the reciprocal of the distance as the original density reliability, and normalize it within the cluster to obtain the density score; the interval score is calculated as follows: calculate the difference between the distance of each sample to the centroid of the assigned cluster and the distance to the centroid of the nearest competing cluster as the original interval, and normalize it within the cluster to obtain the interval score. S5, Quality Perception Comparison Refinement Training After completing the pseudo-label generation and quality-aware sample screening in step S4, supervised contrastive loss is calculated for high-confidence selected samples using pseudo-labels, and semantic alignment loss is superimposed. The encoder is updated in the independent gradient step. For low-confidence unselected samples, the encoder is updated only with instance-level consistency loss among the three modalities. Steps S4 and S5 are executed iteratively until the preset number of refinement rounds is reached, and the final clustering result is output. During inference, the samples to be processed are input into the trained shared multimodal encoder, its fusion representation is extracted, and clustering labels are assigned according to the nearest centroid rule, thereby realizing a pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue data.

2. The pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue as described in claim 1, characterized in that: In step S4, the density-interval joint scoring criterion includes a comprehensive reliability score. The weighted average of the density score and the interval score is shown in equation (2): （2） in, This represents the overall reliability score of the i-th sample. This represents the density score of the i-th sample. This represents the interval score of the i-th sample. This represents the balance coefficient between the density score and the interval score; The samples within each cluster are sorted from highest to lowest based on their overall scores. The top percentage of samples are selected as the high-confidence selected set, while the rest are selected as the low-confidence unselected set. The selection percentage increases incrementally from the initial value with each training round until a preset upper limit is reached, gradually expanding the size of the supervised subset.

3. The pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue as described in claim 2, characterized in that: The balance coefficient is set to 0.5, meaning that the density score and the interval score each account for half of the weight. When the balance coefficient is 1, it degenerates into pure density screening, and when the balance coefficient is 0, it degenerates into pure interval screening. The above calculation is only performed on clusters with a sample size greater than 1, and a neutral score is directly assigned to single-sample clusters.

4. The pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue as described in claim 1, characterized in that: In step S4, pseudo-label generation, the zeroth round uses K-means plus plus sign initialization and performs multiple random restarts to avoid geometric bias; in subsequent rounds, the cluster centroids of the previous round are used as hot start initialization and only a single initialization is performed to reduce computational overhead; K-means clustering is performed on L2 normalized features using Euclidean distance, and the cluster centroids are also L2 normalized before distance evaluation to ensure consistency of the distance metric used in clustering and sample selection.

5. The pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue as described in claim 1, characterized in that: The calculation of the total pre-training loss in step S3: （3） in, Indicates the total pre-training loss, Indicates instance-level modality drop contrast loss, Indicates the nearest neighbor comparison loss across sample texts, This represents the semantic alignment loss of a large multimodal language model. This represents the weights of the nearest neighbor comparison loss across sample texts. This represents the semantic alignment loss weights for large-scale multimodal language models.

6. The pseudo-label-guided semantic clustering method for unlabeled multimodal dialogue as described in claim 1, characterized in that: The training objective for the high-confidence samples in the iterative refinement stage of step S4 consists of two parts. The first part is a supervised contrastive loss using K-means pseudo-labels as class supervision signals, where dropped views of different modalities of the same utterance and other selected samples of the same pseudo-label category are considered positive examples. The second part is a cosine alignment loss with the frozen teacher representation of the large multimodal language model. The encoder is updated in an independent gradient step after the weighted sum of the two losses. The training objective for the low-confidence samples is only the instance-level cross-view consistency loss between dropped views of different modalities of the same utterance, without using any pseudo-label supervision.

Images