A sample clustering method, device and medium based on full-stage uncertainty modeling

Abstract

This invention belongs to the field of artificial intelligence and multimodal learning technology, specifically relating to a sample clustering method, device, and medium based on full-stage uncertainty modeling. This invention achieves unified modeling of multi-dimensional uncertainty across the entire process, simultaneously quantifying three types of uncertainty—data noise, model fluctuation, and prediction confidence—in the complete process of feature extraction, modality fusion, clustering inference, and self-supervised feedback. It connects the credibility constraints of the data layer, model layer, and decision layer, overcoming the shortcomings of traditional methods' segmented and singular credibility modeling. It suppresses representational distortion and clustering bias caused by the cascading transmission of uncertainty throughout the entire process. Furthermore, this invention supports purely unsupervised clustering, eliminating the need for manual sample labeling during operational use, thus saving data labeling costs. The model convergence speed and hyperparameter robustness are moderate, requiring no dedicated expansion of high-performance computing clusters, appropriately reducing hardware maintenance and upgrade expenses.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal learning technology, specifically relating to a sample clustering method, device and medium based on full-stage uncertainty modeling. Background Technology

[0002] With the rapid development of AI, multimodal learning has gradually become an important technical means for handling complex data analysis tasks. A federated multimodal clustering method based on self-supervised multi-scale kernel diffusion graph learning, disclosed in Chinese invention patent application CN122471089A (publication date July 28, 2026), is a sample clustering method based on multimodal data. This method proposes an end-to-cloud collaborative framework: the client uses consistent pseudo-labels issued by the server to calculate the intra- and inter-class scatter matrices, learns the Mahalanobis distance metric, generates a squared anchor graph of the discriminant norm, and uploads it to protect data privacy; the server calculates the graph Laplacian matrix for each modality anchor graph, generates a hot kernel representation on multiple diffusion time scales to filter noise, obtains a consensus kernel matrix through entropy regularization weighted fusion, and then constructs a global consensus graph and updates the pseudo-labels through logarithmic pullback and Laplacian cone projection with sparsity and algebraic connectivity constraints. Iterative optimization continues until convergence, outputting high-quality cross-modal clustering results.

[0003] However, due to differences in acquisition methods, data distribution, and information quality among different modal data, practical applications are often affected by factors such as modal noise, information redundancy, and inconsistencies between modalities. This makes it difficult for the model to accurately determine the effectiveness of information from different modalities, thus affecting the reliability of the model's output. Multimodal learning still faces the following problems in reliable information modeling:

[0004] 1) Relying on manually labeled information for reliability learning is insufficient to meet the data analysis needs of unsupervised scenarios. In multimodal data processing, it is often necessary to use category labels or additional manually labeled information to assist the model in learning the importance and reliability of different modalities. However, in practical applications such as medical analysis, bioinformatics processing, and industrial inspection, large-scale, high-quality labeled data usually relies on professionals, which presents problems such as high acquisition costs, long cycles, and difficulties in expansion. Furthermore, in some complex data scenarios, it is even difficult to obtain effective label information, making it difficult to directly apply reliability modeling methods based on labeled information.

[0005] 2) Reliability modeling typically targets a single processing stage, making it difficult to cover the changes in information reliability throughout the entire multimodal learning process. Multimodal learning usually includes multiple stages such as feature extraction, cross-modal information interaction, feature fusion, and final decision-making. In actual processing, different stages may be affected by noise and uncertainties. However, some technical solutions only perform reliability analysis on a specific stage, such as measuring information quality in the feature representation stage or adjusting the modal contribution level in the fusion stage. This makes it difficult to achieve a unified reliability description throughout the entire process from low-level feature learning to high-level decision output. When unreliable information generated in intermediate stages is not identified and suppressed in a timely manner, it may further affect subsequent fusion results and the final model decision.

[0006] 3) Single types of uncertainty information are insufficient to fully characterize the complex reliability features of multimodal data. Due to the diverse sources of multimodal data, each modality may be affected by factors such as acquisition errors, environmental changes, data gaps, and distribution shifts, resulting in significant differences in their reliability. Furthermore, the effective information contribution of the same modality may vary across different samples. Therefore, multimodal reliability is not only related to the characteristics of the data itself but is also influenced by factors such as model prediction stability and intermodal consistency. Evaluating reliability solely using a single type of uncertainty information is insufficient to comprehensively reflect the potential risk factors in multimodal data, easily leading to inaccurate reliability estimates and insufficient utilization of effective information.

[0007] 4) The lack of a sample-level dynamic reliability adjustment mechanism hinders the full utilization of multimodal complementarity. In real-world multimodal data, different modalities provide complementary information, but their contribution levels are not fixed and change dynamically with variations in sample content and data quality. When some modalities contain noisy information or exhibit modal conflicts, the inability to adaptively adjust based on modal reliability can easily lead to low-quality information participating in model learning, reducing the effectiveness of the fused representation. Furthermore, fixed information fusion strategies struggle to fully exploit the complementary relationships between different modalities, limiting the adaptability of multimodal models in complex scenarios. Summary of the Invention

[0008] The purpose of this invention is to provide a sample clustering method, apparatus and medium based on full-stage uncertainty modeling, in order to solve the problems in the prior art that only use a single type of uncertainty information to evaluate the reliability, which makes it difficult to fully reflect the potential risk factors in multimodal data, and easily leads to inaccurate reliability estimation and insufficient utilization of effective information.

[0009] To address the aforementioned technical problems, the first aspect of this invention provides a sample clustering method based on full-stage uncertainty modeling, comprising: performing the following process on each sample to be clustered to achieve sample clustering:

[0010] 1) For a sample of multimodal data, based on a variational encoder that introduces a random deactivation mechanism, the average latent representation of each mode under different model parameter perturbations is extracted, and the random uncertainty corresponding to each mode is obtained to measure the reliability of the sample and the model uncertainty is obtained to measure the stability of the feature representation under model parameter perturbations.

[0011] 2) Using an uncertainty modeling mechanism based on evidence theory, based on the average latent representation, we obtain the predictive uncertainty for each modality to measure the credibility of clustering decisions, and the trust quality of each modality in each category.

[0012] 3) For a modality, the reliability of the confidence quality of the modality is adjusted based on random uncertainty and model uncertainty; then for a category, the adjusted confidence quality of each modality under the category is fused to obtain the category confidence quality of the category; the category to which the sample belongs is determined based on the category confidence quality of each category; the overall uncertainty is obtained by fusing the prediction uncertainty of each modality to optimize the model parameters.

[0013] In one possible implementation, the process of extracting the average latent representation of a mode under different model parameter perturbations includes:

[0014] The mode is input into the variational encoder corresponding to the mode, which introduces a random deactivation mechanism, to obtain the latent feature representation of the mode under each model parameter perturbation. The average value of the latent feature representation of the mode under each random perturbation is then calculated, which is the average latent representation of the mode under different model parameter perturbations.

[0015] In one possible implementation, the model uncertainty corresponding to a certain mode is calculated as follows:

[0016]

[0017] In the formula, Indicates the first The first sample One modality, express The corresponding average latent representation, express In the The latent feature representation obtained under sub-model parameter perturbation This represents the total number of perturbations to the model parameters. express The corresponding model uncertainty.

[0018] In one possible implementation, the way to obtain the stochastic uncertainty corresponding to a certain mode is as follows:

[0019] The mode is input into a variational encoder that introduces a random deactivation mechanism. The variance parameter of the potential distribution is obtained using the mapping function in the variational encoder. The variance parameter is the random uncertainty corresponding to the mode.

[0020] In one possible implementation, the trust quality for a given modality under each category is obtained as follows:

[0021] The average latent representation of this modality is input into the evidence generation module, which then converts the average latent representation into an evidence vector for the corresponding class using a non-negative mapping function. :

[0022]

[0023] In the formula, Representing modes In the The evidence corresponding to each category , Total number of categories;

[0024] Constructing Dirichlet distribution parameters:

[0025]

[0026] In the formula, Representing modes In the Dirichlet distribution, the first Parameters corresponding to each category;

[0027] Based on the Dirichlet distribution, the trust quality of this modality in each category is obtained using the following formula:

[0028]

[0029] In the formula, Representing modes In category The quality of trust.

[0030] In one possible implementation, the formula for calculating the prediction uncertainty corresponding to this mode is:

[0031]

[0032] In the formula, Representing modes The corresponding forecast uncertainty.

[0033] In one possible implementation, the reliability adjustment method is as follows:

[0034] For a certain mode Find the mean value of the model uncertainty of the mode under each random perturbation, and find the mean value of the random uncertainty of the mode under each random perturbation;

[0035] Fusion mode The mean of the corresponding random uncertainty and the mean of model uncertainty To obtain the mode Corresponding reliability weight Among these, the larger the mean of random uncertainty, the smaller the reliability weight; the larger the mean of model uncertainty, the smaller the reliability weight.

[0036] Reliability weight and Multiply to pair Make reliability adjustments.

[0037] In one possible implementation method The calculation formula is any one of the following:

[0038]

[0039]

[0040]

[0041] In the formula, This represents the total number of modes.

[0042] In one possible approach, the Dempster-Shafer evidence theory is used to fuse the trust quality of each modality within the category, as well as the predictive uncertainty of each modality.

[0043] In one possible implementation, the category corresponding to the highest category confidence quality is selected as the category to which the sample belongs.

[0044] In one possible implementation, when the overall uncertainty is less than the set uncertainty threshold, the sample is considered to have high credibility, and a pseudo-label for the corresponding category is generated for model parameter optimization; when the overall uncertainty is greater than or equal to the set uncertainty threshold, it does not participate in pseudo-label updates.

[0045] To address the aforementioned technical problems, a second aspect of the present invention provides a sample clustering apparatus based on full-stage uncertainty modeling, comprising a processor for executing a computer program to implement the steps of the method in any possible implementation of the first aspect of the present invention.

[0046] To address the aforementioned technical problems, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any possible implementation of the first aspect of the present invention.

[0047] The beneficial effects of this invention are as follows: This invention utilizes a fully unsupervised uncertainty quantification mechanism, acquiring cognitive uncertainty at the model level through multiple random forward propagations, and deriving predictive uncertainty based on evidence theory and Dirichlet distribution. All uncertainty indicators are autonomously generated from the original data, eliminating the need for manual annotation. Simultaneously, by constructing a complete and reliable link covering feature representation, modality fusion, cluster optimization, and feedback adjustment, it leverages random uncertainty to constrain the impact of noisy samples, cognitive uncertainty to enhance the stability of feature representations, and predictive uncertainty to control modality fusion conflicts and pseudo-label screening, achieving coordinated control of risks at the data, model, and prediction levels. The pure unsupervised clustering of this invention eliminates the need for manual sample annotation during the operational phase, saving data annotation costs. The model convergence speed and hyperparameter robustness are moderate; under the same hardware conditions, the computational power consumption from repeated parameter tuning and training is reduced, while data processing throughput is slightly improved. Existing basic computing power equipment can basically handle the computation, eliminating the need for dedicated expansion of high-performance computing clusters and moderately reducing hardware maintenance and upgrade expenses. Attached Figure Description

[0048] Figure 1 This is a diagram of the overall architecture of the model of this invention;

[0049] Figure 2 This is a flowchart of the multimodal data clustering method based on full-stage uncertainty modeling of the present invention;

[0050] Figure 3 This is an architectural diagram of the sample clustering device based on full-stage uncertainty modeling of the present invention. Detailed Implementation

[0051] The core of this invention lies in achieving unified modeling of multi-dimensional uncertainties across the entire process. It simultaneously quantifies various uncertainties, including data noise, model fluctuations, and prediction confidence, throughout the complete process of feature extraction, modality fusion, clustering inference, and self-supervised feedback. This connects the data layer, model layer, and decision layer with reliable constraints, overcoming the shortcomings of traditional methods' segmented and singular reliable modeling. The entire process suppresses representational distortion and clustering bias caused by the cascading transmission of uncertainty. To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description, in conjunction with the accompanying drawings, further illustrates the invention.

[0052] Implementation 1 of a sample clustering method based on full-stage uncertainty modeling:

[0053] This invention addresses four core pain points of existing unsupervised multimodal clustering by proposing a multimodal data clustering method based on full-stage uncertainty modeling. First, existing reliable clustering methods heavily rely on manual annotation. Massive amounts of text and image data, as well as real-world multi-sensor data, lack labels, making manual annotation extremely costly. Existing methods cannot complete complete uncertainty modeling under purely unsupervised conditions, making them unsuitable for scenarios such as unlabeled image retrieval and public opinion clustering. Second, reliable modeling is simplistic, introducing constraints only at the representation or fusion stage, and modeling only one or two of the three types of uncertainty: random, cognitive, and predictive. This fails to cover the entire process from feature extraction, modality fusion, clustering inference, to pseudo-label feedback. Multiple layers of risk from data noise, model parameter fluctuations, and modality evidence conflicts propagate layer by layer, resulting in blurred cluster boundaries and disordered clustering. Third, traditional solutions default to equal weighting of all modalities during fusion, failing to distinguish between high-quality, clear modalities and noisy, incomplete modalities. When modality conflicts occur, low-reliability modality evidence participates in fusion equally, amplifying discrepancies and significantly reducing clustering accuracy. Existing evidence fusion methods do not utilize data and model uncertainties to dynamically adjust modality contributions. Fourth, there is no pseudo-label confidence filtering mechanism. The fuzzy and conflicting sample pseudo-labels generated by unsupervised clustering directly participate in network training, continuously polluting the encoder parameters and causing the latent space cluster structure to degenerate. There is also a lack of a reverse feedback link for prediction confidence, resulting in unidirectional optimization, slow model convergence, and a high risk of getting trapped in local optima.

[0054] This method achieves dynamic perception and reliable fusion of information quality across different modalities by jointly modeling data-level uncertainty, model-level uncertainty, and decision-level uncertainty. The network model architecture is as follows: Figure 1 As shown, the specific implementation process is as follows: Figure 2 As shown, the details are as follows:

[0055] Step 1: Obtain multiple samples, each of which includes multimodal data.

[0056] Multimodal data is input data containing information from multiple modalities. It can include images, text, video, and other forms of data, with different modalities describing information about the same object from different perspectives. The acquired multimodal data can be represented as:

[0057]

[0058] In the formula, Indicates the number of modes. Indicates the first Data for each modality.

[0059] Due to differences in data sources, information representation methods, and observation conditions across different modalities, the data from each modality typically contain varying degrees of noise, missing information, and quality variations. This invention directly inputs the acquired multimodal data into a subsequent uncertainty-aware feature learning module, models the different modalities using a unified probability representation method, and further mines the effective information and potential uncertainties contained in each modality.

[0060] It should be noted that steps two through five are described from the perspective of a single sample. A sample includes images, text, videos, and other forms of multimodal data.

[0061] Step 2: Model stochastic uncertainty.

[0062] To obtain a latent representation that reflects differences in data quality, this invention introduces a variational coding mechanism in the feature learning stage to probabilistically represent each modal input obtained in step one. Unlike traditional variational autoencoders used for generation tasks, this invention only uses the coding part for latent space modeling and does not set up a decoder structure. Its purpose is to learn continuous and structured latent feature representations and to use the variance information in the latent distribution to characterize the uncertainty of the samples themselves.

[0063] Specifically, for the first The first sample Modal input samples First, the initial feature representation is obtained through a modal feature encoding network:

[0064]

[0065] In the formula, Indicates the first Feature encoding network corresponding to each modality Indicates input sample The encoded feature representation.

[0066] Then, the initial feature representation The input variational coding module (i.e., the encoder in the variational autoencoder) is processed through two independent mapping functions. , Obtain the mean and variance parameters of the latent distribution:

[0067]

[0068] In the formula, express The mean value corresponding to each mode. This represents the random uncertainty caused by factors such as data noise, missing information, and observation errors, i.e., the first... The first sample Each mode corresponds to random uncertainty.

[0069] The encoder then outputs the latent feature representations for each sample:

[0070]

[0071] In the formula, For the sample The latent feature representation. This uncertainty is used to measure the reliability of the information of the sample itself. When a certain mode is affected by strong noise, its latent distribution variance increases, indicating that the mode contains more uncertain information; conversely, when the mode data quality is high, its distribution is more concentrated, and the corresponding uncertainty decreases. Through probabilistic representation learning, this invention can capture the uncertain information of the data itself while obtaining effective features.

[0072] Step 3: Model cognitive uncertainty (or model uncertainty).

[0073] After obtaining the latent feature representations in step two, to further characterize the impact of model parameter changes on the stability of feature representations, this invention introduces a random dropout mechanism in the variational coding process. This mechanism estimates model uncertainty by simulating random perturbations of model parameters. Specifically, during the training and inference process of the feature encoding network, a portion of neurons in the network are randomly discarded according to a preset proportion; that is, the outputs of some neurons are set to zero with a certain probability of inactivation. This changes the effective connection structure of the network, causing the model to produce different feature representations under different random model parameter perturbation states. (The last sentence is incomplete and likely refers to a separate step.) During the next random forward propagation, the random mask generated by Dropout is: Then the corresponding latent feature representation can be expressed as:

[0074]

[0075] In the formula, This represents a variational coding network after introducing random deactivation; Indicates network model parameters; Indicates the first Random Dropout mask generated during the next forward propagation; Indicates the first The latent feature representations obtained under sub-random model parameter perturbations. After multiple forward propagations, multiple latent representations of the same sample under different model states can be obtained, thus further calculating the degree of change between latent representations under different random model parameter perturbations:

[0076]

[0077] In the formula, Indicates sample The corresponding average latent representation obtained through multiple random inferences (i.e., under different model parameter perturbations) is the average latent representation under all sub-random model parameter perturbations. We get the average. It represents model uncertainty and is used to measure the stability of feature representations under model parameter perturbations; This indicates the number of perturbations to the random model parameters. Unlike random uncertainty, random uncertainty primarily reflects instability caused by factors such as insufficient model parameter learning, training process fluctuations, and insufficient existing knowledge. When the model's learning for a particular modality sample is insufficient, the feature representations obtained under different random model parameter perturbations differ significantly. As the model's learning ability increases and parameters are optimized more thoroughly, the model output tends to stabilize, and its random uncertainty gradually decreases.

[0078] Step 4: Model and predict uncertainty.

[0079] After completing the feature representation learning in steps two and three, this invention further introduces an uncertainty modeling mechanism based on evidence theory in the clustering stage to quantify the category support information of different modalities.

[0080] Specifically, for the first Each modality represents the average latent representation obtained during the feature learning phase. The input evidence generation module converts the latent representation into evidence vectors for the corresponding categories using a non-negative mapping function (such as the Softplus function). :

[0081]

[0082] In the formula, Representing modes In the The evidence corresponding to each category , This indicates the number of cluster categories.

[0083] And further construct the Dirichlet distribution parameters:

[0084]

[0085] In the formula, In the Dirichlet distribution, the th The parameters corresponding to each category.

[0086] Based on the Dirichlet distribution, the prediction uncertainty can be obtained. and modality Trust quality under each category:

[0087]

[0088] In the formula, Indicates Dirichlet intensity; It represents the uncertainty of prediction and is used to measure the credibility of the current modality clustering decision; Representing modes In category The results indicate that the amount of categorical evidence directly affects the prediction outcome: the more abundant the categorical evidence, the higher the corresponding probability; conversely, when the overall evidence is weak, the uncertainty of the prediction increases significantly.

[0089] Based on this, the present invention incorporates the cognitive uncertainty obtained during the feature learning stage. and random uncertainty Further construct a modal reliability adjustment mechanism to obtain modal reliability. Corresponding reliability weight Reliability weight Primarily used for reliability adjustment before multimodal evidence fusion, its design is based on the source reliability discount concept in evidence theory. Because different modalities are affected by factors such as noise, missing information (corresponding to stochastic uncertainty), and differences in model learning ability (corresponding to cognitive uncertainty), the category evidence they provide does not have the same level of credibility. Before fusion, their contributions need to be dynamically adjusted according to the levels of stochastic and cognitive uncertainty.

[0090] In the invention and It is not solely determined by the samples, but is influenced by the input and the random state of the model. Because a random deactivation mechanism is introduced in step three, different random perturbations will lead to different model parameters, thus potentially yielding different results under different perturbations. and For the same mode, multiple random inactivations will generate multiple sets of uncertainty estimation results, which together reflect the stability of the mode under different model states. This invention averages the uncertainty results obtained from multiple random perturbations and uses them in the subsequent evidence fusion process, i.e., in the subsequent formulas (10) to (12). and It is the mean of random uncertainty under multiple perturbations and the mean of cognitive uncertainty (i.e. model uncertainty).

[0091] Specific construction The method is not unique; it can be in the exponential form shown in formula (10), the reciprocal form shown in formula (11), or the normalized form shown in formula (12). Regardless of the form, the model uncertainty is reflected. and random uncertainty and The relationship between them is: The larger, The smaller; The larger, The smaller.

[0092]

[0093]

[0094]

[0095] In the formula, Representing modes random uncertainty, Indicates the first Cognitive uncertainty in each modality (i.e., model uncertainty) Representing modes The reliability weight is adjusted. When a mode is affected by noise or the model cannot effectively learn the features of that mode, its reliability weight is reduced, thereby reducing the impact of erroneous evidence from that mode on the fusion result.

[0096] This invention, based on the source discounting mechanism in evidence theory, adjusts each modality of evidence using reliability weights:

[0097]

[0098] In the formula, For the adjusted mode Trust quality under each category.

[0099] The adjusted multimodal evidence is then constructed as a quality function:

[0100]

[0101] In the formula, Indicates the first The evidence quality distribution corresponds to each modality. Evidence from different attempts is fused using Dempster-Shafer (DS) theory to generate fused confidence evidence and uncertainty. Subsequently, this invention utilizes Dempster-Shafer evidence theory to fuse the reliability-adjusted multimodal evidence:

[0102]

[0103] In the formula, This represents the global evidence distribution after fusion, including the fused class confidence quality. and overall uncertainty That is, the confidence quality of the fused categories. It is to integrate various modalities in categories Trust quality The overall uncertainty is obtained. It integrates the prediction uncertainties of various modes. By fusing complementary information from different modalities, we can enhance category support among consistent modalities while reducing the propagation of unreliable information caused by modality conflicts.

[0104] Ultimately, this invention utilizes the fused class confidence quality. As the basis for clustering decisions, the category corresponding to the highest confidence quality is selected as the final clustering label for the samples. (i.e., pseudo-tags):

[0105]

[0106] Meanwhile, the uncertainty resulting from the fusion This is used to evaluate the credibility of the current clustering results and further participate in the constraint optimization in step five to reduce conflicts between different modal predictions and improve the stability of multimodal clustering results.

[0107] Step 5: Generate credible pseudo-labels and optimize the model.

[0108] To improve the reliability of pseudo-label supervision information, this invention filters samples based on the uncertainty of the fused predictions. Specifically, this is achieved by setting an uncertainty threshold. (Its specific value can be) For some datasets, no additional adjustments are needed to avoid selection bias caused by repeated parameter tuning. When the samples meet... When the sample is considered to have high credibility, corresponding pseudo-labels are generated to optimize model parameters (specifically, cross-entropy loss can be selected); for High-uncertainty samples are excluded from pseudo-label updates to reduce the impact of noisy supervision on model training, while they are retained for uncertainty regularization constraints to improve model training stability through latent distribution constraints.

[0109] Implementation Method 2 of a Sample Clustering Method Based on Full-Stage Uncertainty Modeling:

[0110] The purpose of this invention is to cluster all the acquired samples (which are multimodal samples) in an unsupervised manner, that is, to divide the samples into clusters. The final result is that some samples are clustered into the first cluster, some samples are clustered into the second cluster, and so on.

[0111] In medical image grouping scenarios, this method can be applied to the auxiliary analysis of patients with brain diseases. For the same patient, multiple sources of data can be collected, including magnetic resonance imaging (MRI), computed tomography (CT) images, and the patient's clinical records. MRI images are mainly used to describe the soft tissue structure of the brain, CT images can provide density characteristics of different tissue regions, and clinical records contain auxiliary information such as patient symptoms and examination results. Due to differences in information representation and reliability among different data sources, some modalities may be affected by image noise, acquisition conditions, or missing information. This method first performs feature representation learning on data from different modalities, and then performs evidence modeling and fusion based on the category information and uncertainty level contained in each modality. This allows reliable modalities to provide more effective information, reduces the interference of low-quality modalities on clustering results, and ultimately achieves automatic grouping of patient samples.

[0112] In agricultural monitoring scenarios using drones, visible light cameras, multispectral sensors, and environmental sensors mounted on the drones can be used to collect regional monitoring data. Visible light images are used to acquire crop appearance and texture features, multispectral images reflect crop spectral variations, and environmental sensor data provides information on the growth environment, such as temperature and humidity. However, the reliability of data from different modes varies due to factors such as changes in illumination and equipment errors during the data acquisition process. To address this issue, this method utilizes complementary information between different modes for joint modeling and adjusts the influence of each mode during the fusion process based on its uncertainty, thereby achieving cluster analysis of different crop states or regional characteristics.

[0113] Specifically, the following process is performed on each sample to achieve clustering:

[0114] Step 1, for a sample Multimodal data Based on a variational encoder that incorporates a random deactivation mechanism, the average latent representation of each mode under different model parameter perturbations is extracted. And obtain the random uncertainty corresponding to each mode for measuring the reliability of the sample. and model uncertainty used to measure the stability of feature representations under model parameter perturbations .

[0115] For the first Modal input samples By using the feature encoding network corresponding to the modality, the initial feature representation corresponding to each modality is obtained. :

[0116]

[0117] In the formula, Indicates the first Feature encoding network corresponding to each modality Indicates input sample The encoded feature representation.

[0118] Represent the initial features The input variational coding module (i.e., the encoder in the variational autoencoder) is processed through two independent mapping functions. , Obtain the mean and variance parameters of the latent distribution:

[0119]

[0120] In the formula, express The mean value corresponding to each mode. It represents random uncertainty caused by factors such as data noise, missing information, and observation errors.

[0121] The encoder then outputs the latent feature representations for each sample:

[0122]

[0123] In the formula, For the sample The latent features are represented.

[0124] The "random deactivation mechanism" estimates model uncertainty by simulating random perturbations of model parameters. During the training and inference of the feature encoding network, a portion of neurons are randomly discarded according to a preset proportion; that is, the outputs of some neurons are set to zero with a certain deactivation probability. This alters the network's effective connection structure, causing the model to produce different feature representations under different random model parameter perturbation states. The latent features generated during the random forward propagation process are represented by the following formula:

[0125]

[0126] In the formula, This represents a variational coding network after introducing random deactivation; Indicates network model parameters; Indicates the first Random Dropout mask generated during the next forward propagation; Indicates the first The latent feature representation obtained under sub-random model parameter perturbation.

[0127] Find the first... The first sample Modal input samples Latent feature representation under various model parameter perturbations The average value is the mode. Average latent representation under different model parameter perturbations .

[0128] Then, the degree of change between latent representations under different perturbations of stochastic model parameters is calculated:

[0129]

[0130] In the formula, Indicates sample The corresponding average latent representation obtained after multiple random inferences (i.e., under different model parameter perturbations); This represents model uncertainty and is used to measure the stability of feature representations under model parameter perturbations. It can also be expressed as... ; This indicates the number of perturbations to the parameters of the random model.

[0131] Step 2: Utilize an uncertainty modeling mechanism based on evidence theory, and calculate the average latent representation corresponding to each modality. This yields the prediction uncertainty for each modality, used to measure the reliability of the current modality clustering decision. and the trust quality of each modality under each category. .

[0132] For the Each modality represents the average latent representation obtained during the feature learning phase. The input evidence generation module converts the latent representation into evidence vectors for the corresponding categories using a non-negative mapping function (such as the Softplus function). :

[0133]

[0134] In the formula, Representing modes In the The evidence corresponding to each category , This indicates the number of cluster categories.

[0135] And further construct the Dirichlet distribution parameters:

[0136]

[0137] In the formula, In the Dirichlet distribution, the th The parameters corresponding to each category.

[0138] Based on the Dirichlet distribution, the prediction uncertainty can be obtained. and modality Trust quality under each category:

[0139]

[0140] In the formula, Indicates Dirichlet intensity; It represents the uncertainty of prediction and is used to measure the credibility of the current modality clustering decision; Representing modes In category The quality of trust.

[0141] Step 3, for a category Based on random uncertainty and model uncertainty Reliability tuning is performed on the trust quality of this mode to obtain... Trust quality adjusted for reliability across all modalities within this category; Obtain the category confidence quality of this category. Determine the category to which the sample belongs based on the category confidence quality of each category; fuse the prediction uncertainties of each modality. The overall uncertainty is obtained To optimize model parameters.

[0142] For a mode Calculate the mean value of the model uncertainty of the mode under each random perturbation, i.e., the mean value in formula (25). And calculate the mean of the random uncertainty of the mode under each random disturbance, that is, the mean of the random uncertainty in formula (25). Modes are generated using two means. Corresponding reliability weight Utilizing reliability weights Reliability tuning is performed to obtain trust quality. :

[0143]

[0144]

[0145] The adjusted multimodal evidence is then constructed as a quality function:

[0146]

[0147] By combining evidence from different attempts using the DS theory, a fused confidence evidence is generated. and uncertainty That is, this invention utilizes the Dempster-Shafer evidence theory to fuse multimodal evidence:

[0148]

[0149] In the formula, This represents the global evidence distribution after fusion, including the fused class confidence quality. and overall uncertainty For a category Specifically, it involves classifying each modality within this category. Trust quality Perform fusion processing to obtain categories Category confidence quality Other categories are also handled in this way. The prediction uncertainties from each modality are fused. The overall uncertainty is obtained .

[0150] Furthermore, the fused category confidence quality As the basis for clustering decisions, the category corresponding to the highest confidence quality is selected as the final clustering label for the samples. (i.e., pseudo-tags):

[0151]

[0152] Meanwhile, the uncertainty resulting from the fusion This is used to evaluate the reliability of the current clustering results and further participate in the constraint optimization in step five to reduce conflicts between different modal predictions and improve the stability of multimodal clustering results. Specifically, this is achieved by setting an uncertainty threshold. ( By comparing overall uncertainty With uncertainty threshold The size relationship between the samples is used to determine whether pseudo-label updates are needed. When the samples satisfy... When the sample is considered to have high credibility, a corresponding pseudo-label is generated and the model parameters are optimized. The pseudo-label update formula is detailed in formula (29). High-uncertainty samples are excluded from pseudo-label updates to reduce the impact of noisy supervision on model training, while they are retained for uncertainty regularization constraints to improve model training stability through latent distribution constraints.

[0153] An implementation method for a sample clustering device based on full-stage uncertainty modeling:

[0154] This invention provides a sample clustering device based on full-stage uncertainty modeling, such as... Figure 3 As shown, the system includes a memory, a processor, a system bus, and a computer program stored in the memory. The processor and memory communicate and exchange data with each other via the system bus. The processor executes the computer program to implement a sample clustering method based on full-stage uncertainty modeling according to the present invention.

[0155] The device can be flexibly deployed according to the scenario. Local workstation deployment is the most intuitive, offering advantages such as physical data isolation, zero network latency, and convenient debugging. However, it is limited by single-machine computing power and storage capacity, making it suitable for rapid verification of small to medium-sized data. It can also be deployed in a private cloud or high-performance computing cluster. This deployment method emphasizes scalability and multi-task concurrency, capable of handling full-stage iterative computation of massive samples through distributed scheduling. However, it requires investment in network infrastructure and enhanced transmission encryption to balance performance and data security. It can also be deployed at the edge (such as in industrial sites or mobile terminals). This deployment prioritizes extremely low response latency and bandwidth conservation, but computing power is severely limited, requiring lightweight algorithm pruning, making it suitable for inference rather than training scenarios.

[0156] When selecting a core processor, the choice is not limited to a single architecture; a heterogeneous trade-off can be made based on the dynamic iteration characteristics of the algorithm. For example, CPUs remain the preferred choice for general use, as their powerful branch prediction, out-of-order execution, and large-capacity cache can efficiently handle frequent conditional jumps and random memory accesses in clustering iterations, making them suitable as main control chips. ARM architecture processors, on the other hand, stand out in large-scale cloud deployments due to their superior performance per watt and cost advantages, making them particularly suitable for power-sensitive large-scale parallel nodes. GPUs / NPUs can serve as coprocessors, accelerating matrix operations when deep feature extraction is required, but are not necessary for purely traditional clustering iterations. Memory selection can be based on needs, building an efficient data flow link between dynamic iteration and static storage to fully support the moderate memory bandwidth and I / O throughput requirements of full-stage uncertainty modeling.

[0157] The core processing steps of this sample clustering method based on full-stage uncertainty modeling include:

[0158] First, input data containing multimodal information is acquired. This multimodal data can include images, text, video, and other forms of data, with different modalities describing the same object from different perspectives. For each modal of data, a corresponding feature encoding network is constructed to map the original multimodal data to the latent space, obtaining the latent feature representations for each modality.

[0159] Secondly, in the feature representation learning stage, a probabilistic representation learning mechanism based on variational autoencoders is introduced to perform distributed modeling of latent features for each modality. Specifically, latent features are represented as Gaussian distributions, and probabilistic features are obtained by learning the mean and variance parameters of the distribution. The mean parameter represents the latent semantic features of the samples, and the variance parameter describes the random uncertainty caused by data noise, observation errors, and differences in sample quality. Simultaneously, to characterize the impact of model parameter changes on latent feature representation, this invention employs a random deactivation mechanism to perform multiple random forward inferences. By statistically analyzing the degree of change in latent feature representation under different perturbation states, cognitive uncertainty is obtained, which is used to describe the uncertainty brought about by changes in model parameters and model structure.

[0160] Subsequently, in the clustering reasoning stage, the obtained latent features are input into the evidence generation module. Category evidence information corresponding to each modality is obtained through category evidence mapping and Dirichlet distribution modeling, and prediction uncertainty is further calculated to describe the degree of uncertainty in the clustering reasoning process. Furthermore, combining the random uncertainty and cognitive uncertainty obtained in the feature representation learning stage, an uncertainty-aware modality reliability adjustment mechanism is constructed. Modality reliability weights are adaptively calculated based on the uncertainty levels of different modalities, and these weights are used to dynamically adjust the evidence information generated by each modality. Based on this, a multimodal fusion method based on evidence theory is used to fuse the reliability-adjusted multimodal evidence to obtain the final clustering result.

[0161] Finally, this invention utilizes prediction uncertainty to screen for sample reliability and constructs a reliable pseudo-label optimization mechanism. When the prediction uncertainty of a sample is below a preset threshold, the corresponding clustering result is used as a reliable pseudo-label for further optimization of model parameters; when a sample has high prediction uncertainty, it is avoided from participating in pseudo-label updates to reduce the impact of unreliable information on the model training process.

[0162] For more detailed information on this method, please refer to the descriptions in Implementation 1 and Implementation 2 of a Sample Clustering Method Based on Full-Stage Uncertainty Modeling.

[0163] One embodiment of a computer-readable storage medium:

[0164] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a sample clustering method based on full-stage uncertainty modeling. For more detailed information on this method, please refer to the descriptions in Embodiment 1 and Embodiment 2 of the sample clustering method based on full-stage uncertainty modeling.

[0165] In summary, the present invention has the following characteristics:

[0166] From a social perspective, this invention is adaptable to unlabeled data scenarios such as drone monitoring, image and text retrieval, medical image grouping, and educational question bank classification. Ordinary small and medium-sized enterprises and university laboratories can deploy it using conventional hardware, lowering the barrier to entry for using reliable multimodal clustering technology. By identifying conflicts and noisy samples based on uncertainty, it can help staff distinguish between reliable and questionable clustering results, reducing the probability of making unreasonable business judgments based on distorted clustering conclusions, and providing a more reliable analysis tool for data governance in multiple industries.

[0167] From an economic perspective, this invention supports pure unsupervised clustering, eliminating the need for manpower to label samples during business use and saving data labeling-related costs. The model convergence speed and hyperparameter robustness are moderate. Under the same hardware conditions, the computational power consumption caused by repeated parameter tuning and training is reduced, and the data processing throughput is slightly improved. Existing basic computing power equipment can basically handle the computation, eliminating the need for dedicated expansion of high-performance computing clusters and moderately reducing hardware maintenance and upgrade expenses.

[0168] From a technical perspective, this invention achieves unified modeling of multi-dimensional uncertainties across the entire process. It simultaneously quantifies three types of uncertainties—data noise, model fluctuation, and prediction confidence—in the complete process of feature extraction, modality fusion, clustering inference, and self-supervised feedback. It breaks down the credibility constraints of the data layer, model layer, and decision layer, and solves the shortcomings of traditional methods in terms of segmented and singular credibility modeling. It also suppresses the representation distortion and clustering bias caused by the layer-by-layer propagation of uncertainty throughout the entire process.

Claims

1. A sample clustering method based on full-stage uncertainty modeling, characterized in that, include: The following procedure is performed on each sample to be clustered to achieve sample clustering: 1) For a sample of multimodal data, based on a variational encoder that introduces a random deactivation mechanism, the average latent representation of each mode under different model parameter perturbations is extracted, and the random uncertainty corresponding to each mode is obtained to measure the reliability of the sample and the model uncertainty is obtained to measure the stability of the feature representation under model parameter perturbations. 2) Using an uncertainty modeling mechanism based on evidence theory, based on the average latent representation, we obtain the predictive uncertainty for each modality to measure the credibility of clustering decisions, and the trust quality of each modality in each category. 3) For a modality, the reliability of the trust quality of the modality is adjusted based on random uncertainty and model uncertainty; then for a category, the adjusted trust quality of each modality under the category is fused to obtain the category confidence quality of the category. The category to which the sample belongs is determined based on the category confidence quality of each category; the overall uncertainty is obtained by fusing the prediction uncertainties of each modality to optimize the model parameters.

2. The sample clustering method based on full-stage uncertainty modeling according to claim 1, characterized in that, The process of extracting the average latent representation of a mode under different model parameter perturbations includes: The mode is input into the variational encoder corresponding to the mode, which introduces a random deactivation mechanism, to obtain the latent feature representation of the mode under each model parameter perturbation. The average value of the latent feature representation of the mode under each random perturbation is then calculated, which is the average latent representation of the mode under different model parameter perturbations.

3. The sample clustering method based on full-stage uncertainty modeling according to claim 2, characterized in that, The uncertainty of a model corresponding to a certain mode is calculated as follows: In the formula, Indicates the first The first sample One modality, express The corresponding average latent representation, express In the The latent feature representation obtained under sub-model parameter perturbation This represents the total number of perturbations to the model parameters. express The corresponding model uncertainty.

4. The sample clustering method based on full-stage uncertainty modeling according to claim 1, characterized in that, The method for obtaining the random uncertainty corresponding to a certain mode is as follows: The mode is input into a variational encoder that introduces a random deactivation mechanism. The variance parameter of the potential distribution is obtained using the mapping function in the variational encoder. The variance parameter is the random uncertainty corresponding to the mode.

5. The sample clustering method based on full-stage uncertainty modeling according to claim 1, characterized in that, The method for obtaining the trust quality of a certain modality under each category is as follows: The average latent representation of this modality is input into the evidence generation module, which then converts the average latent representation into an evidence vector for the corresponding class using a non-negative mapping function. : In the formula, Representing modes In the The evidence corresponding to each category , Total number of categories; Constructing Dirichlet distribution parameters: In the formula, Representing modes In the Dirichlet distribution, the first Parameters corresponding to each category; Based on the Dirichlet distribution, the trust quality of this modality in each category is obtained using the following formula: In the formula, Representing modes In category The quality of trust under the circumstances; The formula for calculating the prediction uncertainty corresponding to this mode is as follows: In the formula, Representing modes The corresponding forecast uncertainty.

6. The sample clustering method based on full-stage uncertainty modeling according to claim 5, characterized in that, The reliability adjustment method is as follows: For a certain mode Find the mean value of the model uncertainty of the mode under each random perturbation, and find the mean value of the random uncertainty of the mode under each random perturbation; Fusion mode The mean of the corresponding random uncertainty and the mean of model uncertainty To obtain the mode Corresponding reliability weight Among these, the larger the mean of random uncertainty, the smaller the reliability weight; the larger the mean of model uncertainty, the smaller the reliability weight. Reliability weight and Multiply to pair Make reliability adjustments.

7. The sample clustering method based on full-stage uncertainty modeling according to claim 6, characterized in that, The calculation formula is any one of the following: In the formula, This represents the total number of modes.

8. The sample clustering method based on full-stage uncertainty modeling according to claim 1, characterized in that, When the overall uncertainty is less than the set uncertainty threshold, the sample is considered to have high credibility, and a pseudo-label for the corresponding category is generated for model parameter optimization; when the overall uncertainty is greater than or equal to the set uncertainty threshold, it does not participate in pseudo-label updates.

9. A sample clustering device based on full-stage uncertainty modeling, comprising a processor, characterized in that, The processor executes a computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images