An artificial intelligence-based multi-modal data set cleaning and labeling method

Abstract

The application relates to the technical field of multi-modal data labeling, and specifically discloses a multi-modal data set cleaning and labeling method based on artificial intelligence, which comprises the following steps: acquiring multi-modal data and initial labeling; respectively extracting cognitive uncertainty of image, text and audio modalities and constructing a labeling quality hyperspace; calculating the mean value in each group, and sequentially calculating an image-text cross-modal propagation attenuation rate, a text-audio cross-modal propagation attenuation rate and a pollution propagation asymmetry index; marking the group with an asymmetry index higher than a threshold value as a pollution community, tracking the propagation path and determining a pollution source node, and distinguishing between contaminated samples and pollution sources to be removed; re-labeling the contaminated samples, returning them to the warehouse or removing them, permanently removing the pollution sources, and obtaining a cleaned data set. Through pollution tracing and hierarchical treatment, the application realizes accurate cleaning of multi-modal labeled data.

Description

Technical Field

[0001] This invention relates to the field of multimodal data annotation technology, and in particular to a method for cleaning and annotating multimodal datasets based on artificial intelligence. Background Technology

[0002] Multimodal automatic annotation technology, which automatically annotates multimodal data such as images, text, and audio by calling pre-trained visual-language models (such as CLIP and SAM), has become the mainstream method for constructing large-scale training datasets. However, due to the inherent difficulties in cross-modal semantic alignment and the subjective judgment differences involved in the annotation process, the automatic annotation results generally suffer from the problem of "noisy annotations" where the labeled categories are inconsistent with the content of the multimodal data. Studies have shown that the proportion of label noise in mainstream public datasets can reach 0.15% to 40%, and the performance of models trained on datasets with noisy labels will be significantly reduced.

[0003] Existing technologies attempt to filter low-quality labeled samples through multimodal confidence scoring or uncertainty modeling, but they typically use a globally uniform threshold for batch filtering or direct relabeling, failing to distinguish between the source of noise propagation and secondary contamination samples, nor to block the spread of noise in the dataset at the propagation path level. This "one-size-fits-all" cleaning approach results in the wrong deletion of a large number of secondary contamination samples that could have been repaired through secondary labeling, while the real source of contamination samples remain in the dataset because they are not identified, making the cleaning effect extremely limited. At the same time, existing methods lack modeling of the propagation patterns of noise among labeled samples, making it difficult to fundamentally eliminate contamination sources and prevent the continued spread of noise.

[0004] Therefore, this invention proposes a method for cleaning and labeling multimodal datasets based on artificial intelligence. Summary of the Invention

[0005] This invention provides a method for cleaning and labeling multimodal datasets based on artificial intelligence. By tracing the source of pollution and handling it hierarchically, it solves the shortcomings of existing technologies that cannot distinguish between pollution sources and secondary pollution and that the cleaning is incomplete, thus achieving accurate and efficient cleaning of labeled datasets.

[0006] This invention provides a method for cleaning and labeling multimodal datasets based on artificial intelligence, comprising: Obtain the original multimodal dataset containing image, text, and audio data, call the multimodal automatic annotation model to perform initial annotation, and obtain the initial annotation results for each annotated sample; The cognitive uncertainty of each labeled sample in the three modalities is extracted respectively, and the labeling quality hyperspace is constructed with the three cognitive uncertainties as coordinate axes; All labeled samples were grouped by data type, and the within-group mean of each group was calculated for the three cognitive uncertainties. The image-text cross-modal propagation attenuation rate, the text-audio cross-modal propagation attenuation rate, and the pollution propagation asymmetry index were calculated in turn. Groups with a pollution propagation asymmetry index higher than a preset threshold are marked as polluted communities. The image-text cross-modal propagation attenuation rate and the text-audio cross-modal propagation attenuation rate are used as the basis for calculating the connection edge weights during path tracing. The uncertainty propagation path is traced along the direction of decreasing cross-modal propagation attenuation rate. The labeled sample with the largest attenuation change rate in the labeled quality hyperspace on the path is identified as the pollution source node. The labeled samples in the polluted community other than the pollution source node are marked as polluted samples. The pollution source node is marked as a pollution source to be removed. The contaminated samples are separated from the original multimodal dataset to form the dataset to be cleaned. The dataset to be cleaned is relabeled. The contaminated samples whose cognitive uncertainty is reduced after relabeling are returned to the original multimodal dataset. The contaminated samples whose cognitive uncertainty is not reduced after relabeling are removed from the original multimodal dataset. The contaminants to be removed are permanently removed from the original multimodal dataset to obtain the cleaned multimodal labeled dataset.

[0007] Furthermore, the cognitive uncertainty of each labeled sample across the three modalities is extracted, including: Image data is input into a pre-trained visual perception model. After performing image recognition and image detection on the image data, the visual perception model outputs the first probability distribution of the image data belonging to each category. The first information entropy is calculated based on the first probability distribution and used as the first cognitive uncertainty of the image data. Text data is input into a pre-trained text encoding model. After performing semantic segmentation and semantic classification on the text data, the text encoding model outputs the second probability distribution of the text data belonging to each category. The second information entropy is calculated based on the second probability distribution and used as the second cognitive uncertainty of the text data. The audio data is input into a pre-trained audio coding model. The audio coding model outputs the third probability distribution of the audio data belonging to each category. The third information entropy is calculated based on the third probability distribution and used as the third cognitive uncertainty of the audio data.

[0008] Furthermore, all labeled samples are grouped by data type, including: The labeled samples with image data type are divided into image labeled sample groups; The labeled samples with text data type are divided into text labeled sample groups; The labeled samples with audio data type are divided into audio labeled sample groups; Based on the data source as the dividing attribute, the labeled samples are divided into three groups: the first source labeled sample group, the second source labeled sample group, and the third source labeled sample group. Based on the label category, the labeled samples are divided into three categories: Category 1, Category 2, and Category 3.

[0009] Furthermore, the uncertainty propagation path is traced along the direction of decreasing cross-modal propagation attenuation rate, including: The group of labeled samples with the highest pollution transmission asymmetry index in the polluted community was used as the reference group, and the group of labeled samples with the highest ratio of image modal cognitive uncertainty to text modal cognitive uncertainty in the reference group was used as the starting point for tracking. For any two labeled samples within the polluted community, the cognitive uncertainty of the two labeled samples in the image modality is extracted. The absolute value of the difference between the cognitive uncertainties of the two labeled samples in the image modality is used as the basic weight. The basic weight is multiplied by the absolute value of the image-text cross-modal propagation attenuation rate of the reference group to obtain the image-text connection edge weight between the two labeled samples. The cognitive uncertainty of the two labeled samples in the text modality is extracted. The absolute value of the difference between the cognitive uncertainties of the two labeled samples in the text modality is used as the basic weight. The basic weight is multiplied by the absolute value of the text-audio cross-modal propagation attenuation rate of the reference group to obtain the text-audio connection edge weight between the two labeled samples. The image-text connection edge weight and the text-audio connection edge weight are added together to obtain the fused connection edge weight between the two labeled samples. The fused connection edge weight is used as the edge weight to establish a directed connection edge between the labeled samples within the polluted community. The direction of the directed connection edge is from the labeled sample with the higher fused connection edge weight to the labeled sample with the lower fused connection edge weight. Starting from the tracking point, it iterates along the direction of the directed connection edge until a labeled sample with a fused connection edge weight lower than the preset fused connection edge weight threshold is reached. All labeled samples passed during the traversal are then arranged sequentially as uncertainty propagation paths.

[0010] Furthermore, the labeled samples with the largest rate of decay change in the labeled quality hyperspace along the path are identified as pollution source nodes, including: Calculate the difference between the fusion connection edge weight of each labeled sample and the fusion connection edge weight of the previous labeled sample on the uncertainty propagation path, divide the difference by the Euclidean distance between two adjacent labeled samples on the uncertainty propagation path in the labeled quality hyperspace, and obtain the decay rate at the corresponding labeled sample position. Compare the decay change rate at all labeled sample locations, and select the labeled sample with the largest decay change rate as the pollution source node.

[0011] Furthermore, contaminated samples are separated from the original multimodal dataset to form a dataset to be cleaned, including: Remove the labeled category tags of contaminated samples from the training sample set of the multimodal automatic labeling model; In the cognitive uncertainty propagation network, the edge weights of the connections between contaminated samples and other labeled samples are set to zero. The contaminated samples are marked with isolation tags and stored in an isolation sample library, which is physically separated from the original multimodal dataset.

[0012] Furthermore, the dataset to be cleaned is re-labeled, including: Read contaminated samples with attached isolation tags from the isolated sample library; The multimodal automatic annotation model is invoked to perform secondary annotation on the contaminated samples, and the secondary annotation results are obtained. Extract the secondary cognitive uncertainty of each contaminated sample from the secondary annotation results; When the secondary cognitive uncertainty of a contaminated sample is lower than the cognitive uncertainty in the initial labeling results, the contaminated sample is marked as a cleaned sample, and the cleaned sample is removed from the isolated sample library and re-integrated into the original multimodal dataset. When the secondary cognitive uncertainty of a contaminated sample is not lower than the cognitive uncertainty in the initial labeling results, the contaminated sample is marked as a persistent contaminated sample, and the persistent contaminated sample is kept in the isolated sample library and a manual review is triggered.

[0013] Furthermore, it also includes: Calculate the contamination concentration within each labeled sample group. The contamination concentration within a group is the ratio of the number of contaminated samples in the labeled sample group to the total number of labeled samples in the current labeled sample group. The product of the pollution propagation asymmetry index and the pollution concentration within each labeled sample group is used as the pollution propagation potential of the current labeled sample group. The pollution propagation potential of all labeled sample groups is normalized to obtain the normalized pollution propagation potential of each labeled sample group. The labeled sample groups whose normalized contamination propagation potential energy is higher than the preset potential energy threshold are marked as priority cleaning groups, and the contaminated samples in the priority cleaning groups are re-labeled first.

[0014] Furthermore, it also includes: In each labeled sample group, the ratio of the pollution propagation asymmetry index to the pollution concentration within the group is calculated to obtain the pollution source confidence of the current labeled sample group; The labeled sample groups with pollution source confidence levels higher than the preset confidence threshold are marked as high-confidence pollution groups; In the high-confidence pollution group, the contaminated sample with the highest total cognitive uncertainty value is marked as a candidate pollution source sample; Calculate the label deviation between the label category labels of the pollution source candidate samples and the label category labels of other polluted samples in the high-confidence pollution group. The cosine distance between the label vector corresponding to the label category labels of the pollution source candidate samples and the mean vector of the label vector corresponding to the label category labels of other polluted samples in the high-confidence pollution group is used as the label deviation. When the label deviation exceeds a preset deviation threshold, the candidate samples of the pollution source will be permanently removed from the original multimodal dataset.

[0015] Furthermore, it also includes: In a cognitive uncertainty propagation network, calculate the propagation path length from each contaminated sample in a contaminated community to the source node. Based on the propagation path lengths of all contaminated samples, the distribution of propagation path lengths in the contaminated community is constructed. When the skewness coefficient of the propagation path length distribution exceeds the preset skewness threshold, it is determined that there is multi-source pollution in the polluted community, and the pollution source node localization is re-executed within the polluted community until all pollution source nodes are located. All identified pollution source nodes are permanently removed from the original multimodal dataset as pollution sources to be removed.

[0016] The beneficial effects of this invention compared to existing technologies are as follows: By combining cognitive uncertainty extraction with deep learning, image recognition and detection are performed on image data using a pre-trained visual perception model, outputting probability distributions for each category and calculating information entropy, thus achieving accurate identification of contaminated samples in multimodal labeled data; semantic segmentation and semantic classification are performed on text data using a pre-trained text encoding model, quantifying the distribution characteristics of labeled noise in text modalities; by constructing a cross-modal semantic propagation attenuation model, the source of contamination and secondary contaminated samples are distinguished using the asymmetric index of contamination propagation, solving the deficiency of existing technologies in tracing and locating contamination source nodes; through directed propagation path tracking and attenuation change rate calculation, accurate location of contamination source nodes within the contaminated community is achieved, and contamination source nodes and contaminated samples are treated in a hierarchical manner, eliminating the source of contamination from the root while retaining secondary contaminated samples that can be repaired through secondary labeling, effectively improving the overall quality of the cleaned multimodal labeled dataset and the reliability of subsequent deep learning model training.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the multimodal dataset cleaning and annotation method based on artificial intelligence in an embodiment of the present invention; Figure 2 This is a flowchart of the submodal cognitive uncertainty extraction process in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the division of polluted communities and the location of pollution sources in an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] refer to Figure 1 This invention provides an embodiment of a multimodal dataset cleaning and annotation method based on artificial intelligence, comprising: Obtain the original multimodal dataset containing image, text, and audio data, call the multimodal automatic annotation model to perform initial annotation, and obtain the initial annotation results for each annotated sample; The cognitive uncertainty of each labeled sample in the three modalities is extracted respectively, and the labeling quality hyperspace is constructed with the three cognitive uncertainties as coordinate axes; All labeled samples were grouped by data type, and the within-group mean of each group was calculated for the three cognitive uncertainties. The image-text cross-modal propagation attenuation rate, the text-audio cross-modal propagation attenuation rate, and the pollution propagation asymmetry index were calculated in turn. Groups with a pollution propagation asymmetry index higher than a preset threshold are marked as polluted communities. The image-text cross-modal propagation attenuation rate and the text-audio cross-modal propagation attenuation rate are used as the basis for calculating the connection edge weights during path tracing. The uncertainty propagation path is traced along the direction of decreasing cross-modal propagation attenuation rate. The labeled sample with the largest attenuation change rate in the labeled quality hyperspace on the path is identified as the pollution source node. The labeled samples in the polluted community other than the pollution source node are marked as polluted samples. The pollution source node is marked as a pollution source to be removed. The contaminated samples are separated from the original multimodal dataset to form the dataset to be cleaned. The dataset to be cleaned is relabeled. The contaminated samples whose cognitive uncertainty is reduced after relabeling are returned to the original multimodal dataset. The contaminated samples whose cognitive uncertainty is not reduced after relabeling are removed from the original multimodal dataset. The contaminants to be removed are permanently removed from the original multimodal dataset to obtain the cleaned multimodal labeled dataset.

[0022] In this embodiment, obtaining the raw multimodal dataset containing image, text and audio data refers to obtaining the raw data resources to be cleaned from the data storage medium or data acquisition device. Each data sample in the raw data resource has data content in three modalities: image, text and audio. For example, a product image, the text description associated with the image and a corresponding audio description together constitute a raw multimodal data sample.

[0023] In this embodiment, the multimodal automatic annotation model refers to a pre-trained deep learning neural network model. The model consists of three sub-networks: an image encoder, a text encoder, and an audio encoder, as well as a cross-modal fusion classification layer. During model training, the three encoders are jointly trained using labeled multimodal samples, enabling the model to receive image, text, and audio data as input. After the encoders extract the features of each modality, the fusion classification layer outputs the probability value of the data sample belonging to each category in the preset category set.

[0024] In this embodiment, calling the multimodal automatic annotation model to perform initial annotation means simultaneously inputting each data sample in the original multimodal dataset into the image encoder, text encoder, and audio encoder of the multimodal automatic annotation model. After calculation, the model outputs the probability distribution of the data sample belonging to each category in the preset category set. The category with the highest probability value is taken as the initial annotation category label of the data sample, and the initial annotation category label is the initial annotation result.

[0025] In this embodiment, a labeled sample refers to an independent data unit in the original multimodal dataset. This data unit contains information content from three modalities: image data, text data, and audio data. This data unit corresponds to an initial labeled category label. For example, a product image, the text description associated with the image, and a corresponding audio description together constitute a labeled sample.

[0026] In this embodiment, the three modalities refer to three different types of data representation: image modality, text modality, and audio modality. The image modality carries visual information in the form of a pixel matrix, the text modality carries language information in the form of a character sequence, and the audio modality carries sound information in the form of a waveform sampling sequence.

[0027] In this embodiment, extracting the cognitive uncertainty of each labeled sample across the three modalities means that for each labeled sample, the image data in the labeled sample is input into the image encoder part of the multimodal automatic annotation model to obtain the probability distribution of the image data belonging to each category, and the information entropy value of the probability distribution is used as the cognitive uncertainty of the labeled sample in the image modality; the text data in the labeled sample is input into the text encoder part of the multimodal automatic annotation model to obtain the probability distribution of the text data belonging to each category, and the information entropy value of the probability distribution is used as the cognitive uncertainty of the labeled sample in the text modality; the audio data in the labeled sample is input into the audio encoder part of the multimodal automatic annotation model to obtain the probability distribution of the audio data belonging to each category, and the information entropy value of the probability distribution is used as the cognitive uncertainty of the labeled sample in the audio modality. Cognitive uncertainty reflects the degree of certainty of the model's classification decision for the labeled sample in the corresponding modality. The higher the entropy value, the more uncertain the model's classification result for that modality.

[0028] In this embodiment, constructing a labeling quality hyperspace using three cognitive uncertainties as coordinate axes refers to constructing a three-dimensional geometric space with the first, second, and third cognitive uncertainties as three mutually perpendicular coordinate axis directions. The positive direction of each coordinate axis in this space represents the direction of increasing uncertainty. This three-dimensional geometric space is the labeling quality hyperspace. Each point in this space represents the combination state of cognitive uncertainty of a label sample in the three modalities. This space provides a unified geometric metric framework for subsequent calculation of the distance between label samples.

[0029] In this embodiment, the data type refers to the modal category to which the data content in the labeled sample belongs, including image data type, text data type and audio data type. The data type of each labeled sample is determined according to the modal affiliation of its data content.

[0030] In this embodiment, grouping all labeled samples by data type means that all labeled samples are assigned to corresponding groups according to their respective data types. Labeled samples with image data type are assigned to the image labeled sample group, labeled samples with text data type are assigned to the text labeled sample group, and labeled samples with audio data type are assigned to the audio labeled sample group. All labeled samples in each group have the same data type.

[0031] In this embodiment, calculating the within-group mean for each group across the three cognitive uncertainties refers to, for the image annotation sample group, text annotation sample group, and audio annotation sample group respectively, calculating the arithmetic mean of the cognitive uncertainties of all annotation samples in the image modality within the image annotation sample group, and using this as the within-group mean of the image annotation sample group in the image modality; calculating the arithmetic mean of the cognitive uncertainties of all annotation samples in the text modality within the text annotation sample group, and using this as the within-group mean of the text annotation sample group in the text modality; and calculating the arithmetic mean of the cognitive uncertainties of all annotation samples in the audio modality within the audio annotation sample group, and using this as the within-group mean of the audio annotation sample group in the audio modality.

[0032] In this embodiment, the sequential calculation of the image-text cross-modal propagation attenuation rate, the text-audio cross-modal propagation attenuation rate, and the contamination propagation asymmetry index involves: first, for the image-annotated sample group, subtracting the within-group mean of the image-annotated sample group in the image modality from the within-group mean of the image-annotated sample group in the text modality, and dividing the result by the sum of the two within-group means to obtain the image-text cross-modal propagation attenuation rate; then, for the text-annotated sample group, subtracting the within-group mean of the text-annotated sample group in the text modality from the within-group mean of the text-annotated sample group in the audio modality, and dividing the result by the sum of the two within-group means to obtain the text-audio cross-modal propagation attenuation rate; finally, subtracting the text-audio cross-modal propagation attenuation rate from the image-text cross-modal propagation attenuation rate, taking the absolute value, and then dividing it by the sum of the absolute values ​​of the image-text cross-modal propagation attenuation rate and the text-audio cross-modal propagation attenuation rate to obtain the contamination propagation asymmetry index. This index is used to measure the degree of asymmetry in the propagation direction of cognitive uncertainty between different modalities.

[0033] In this embodiment, the preset threshold is a value set by the operator in advance according to the actual application scenario and accuracy requirements. This value is used to determine whether the pollution propagation asymmetry index of the labeled sample group reaches the standard that it needs to be marked as a polluted community. When the pollution propagation asymmetry index is higher than the preset threshold, it indicates that the labeled sample group has a significant cross-modal uncertainty propagation asymmetry phenomenon and needs to be marked as a polluted community for subsequent processing. The value range of the preset threshold is usually between 0 and 1, and the operator can flexibly adjust it according to the noise level of the dataset.

[0034] It should be noted that the image-text cross-modal propagation attenuation rate, text-audio cross-modal propagation attenuation rate, and contamination propagation asymmetry index involved in the above calculations are all group-level attributes calculated at the labeled sample group level. In the subsequent path tracing steps, each labeled sample within the contamination community inherits the image-text cross-modal propagation attenuation rate and text-audio cross-modal propagation attenuation rate values ​​of its respective labeled sample group, which are used as the group-level attenuation rate attribute value for that labeled sample. When constructing directed connections, the specific calculation method for the edge weights is as follows: the absolute value of the difference between the cognitive uncertainties of two labeled samples in the corresponding modality within the contamination community is used as the basic weight, and this basic weight is multiplied by the cross-modal propagation attenuation rate in the corresponding direction to obtain the connection edge weight between the two labeled samples. Since the cognitive uncertainties of different labeled samples are different, and the absolute values ​​of the differences between each pair of samples are also different, even when multiplied by the same group-level attenuation rate factor, the connection edge weights between the labeled samples within the contamination community are still different from each other, possessing the distinguishability for constructing directed decreasing paths. The starting point for tracking is determined as follows: within the group of labeled samples with the highest asymmetric index of pollution propagation, the ratio between the cognitive uncertainty of the image mode and the cognitive uncertainty of the text mode for each labeled sample is calculated. The labeled sample with the largest ratio is taken as the starting point for tracking. The largest ratio means that the labeled sample has the largest difference in cognitive uncertainty between the image mode and the text mode, and is the most likely starting position for cross-modal uncertainty propagation.

[0035] In this embodiment, the image-text cross-modal propagation attenuation rate and the text-audio cross-modal propagation attenuation rate are used as the basis for calculating the edge weights during path tracing. Tracing the uncertainty propagation path along the direction of decreasing cross-modal propagation attenuation rate means taking each labeled sample in the image labeled sample group as a node. Between any two labeled samples in the image labeled sample group, the absolute value of the difference in cognitive uncertainty of the two labeled samples in the image modality is used as the basic weight. This basic weight is multiplied by the image-text cross-modal propagation attenuation rate to obtain the edge weight between the two labeled samples, thereby constructing a directed network. In this directed network, starting from the labeled sample with the highest image-text cross-modal propagation attenuation rate, adjacent nodes are visited sequentially along the direction of decreasing edge weights. The sequence of nodes visited is taken as the uncertainty propagation path.

[0036] In this embodiment, identifying the label sample with the largest decay rate in the label quality hyperspace as the pollution source node means that for each label sample on the uncertainty propagation path except the starting point, the difference between the fusion connection edge weight of the label sample and the fusion connection edge weight of the previous label sample is calculated. This difference is then divided by the Euclidean distance between the label sample and the previous label sample in the label quality hyperspace to obtain the decay rate at that label sample. The decay rates of all label samples on the path are compared, and the label sample with the largest decay rate is identified as the pollution source node.

[0037] In this embodiment, marking all labeled samples in the polluted community except for the pollution source node as polluted samples means marking all other labeled samples in the polluted community except for the labeled sample that is identified as the pollution source node as polluted samples. The pollution source node is the starting source that causes uncertainty to propagate within the polluted community, while the polluted samples are those whose uncertainty increases due to the influence of the pollution source on the propagation path.

[0038] In this embodiment, separating contaminated samples from the original multimodal dataset to form the dataset to be cleaned means removing all contaminated samples from the original multimodal dataset and taking the set of all the removed contaminated samples as an independent data set, which is the dataset to be cleaned. The removed contaminated samples are no longer retained in the original multimodal dataset, and the contaminated samples are stored centrally for subsequent relabeling processing.

[0039] In this embodiment, relabeling the dataset to be cleaned means re-inputting each contaminated sample in the dataset to be cleaned into the multimodal automatic labeling model. The model re-outputs the probability distribution of each contaminated sample belonging to each category, and uses the category with the highest probability value as the secondary labeling category of the contaminated sample. At the same time, the model outputs the secondary cognitive uncertainty of the contaminated sample in the three modalities.

[0040] In this embodiment, returning contaminated samples with reduced cognitive uncertainty after relabeling to the original multimodal dataset means that for contaminated samples in the dataset to be cleaned, after relabeling, the secondary cognitive uncertainty of the contaminated sample in the three modalities is compared with the initial cognitive uncertainty of the contaminated sample in the three modalities. When the secondary cognitive uncertainty of the contaminated sample in the three modalities is lower than the corresponding initial cognitive uncertainty, the contaminated sample and its corresponding secondary labeling category label are re-stored into the original multimodal dataset, making it a component of the original multimodal dataset.

[0041] In this embodiment, removing contaminated samples whose cognitive uncertainty has not decreased after relabeling from the original multimodal dataset means that for contaminated samples in the dataset to be cleaned that have been relabeled, if the secondary cognitive uncertainty of the contaminated sample in any mode is not lower than the initial cognitive uncertainty corresponding to that mode, the contaminated sample is completely deleted from the entire dataset, and its corresponding initial labeling category label and secondary labeling category label are no longer retained.

[0042] In this embodiment, the cleaned multimodal labeled dataset refers to the set of all remaining labeled samples in the original multimodal dataset obtained after a series of processing steps, including separating contaminated samples, relabeling them, returning them to the database, and removing samples that have not been reduced. Each labeled sample in this set has a verified label category label, and the cognitive uncertainty in each modality is within an acceptable range.

[0043] The beneficial effects of the above technical solutions are as follows: A complete closed loop from pollution identification and source tracing to hierarchical cleaning is constructed, enabling automatic identification, source tracing, and hierarchical disposal of contaminated samples in multimodal datasets. By extracting the cognitive uncertainties of the three modalities to construct a labeling quality hyperspace, quality assessment is transformed into a geometric measurement problem; by calculating the cross-modal propagation attenuation rates and the pollution propagation asymmetry index, the propagation law of uncertainty between modalities is accurately and quantitatively described; by marking contaminated communities, tracing propagation paths, and locating pollution source nodes, the shortcomings of existing technologies in distinguishing between pollution sources and secondary pollution are solved; by relabeling contaminated samples and diverting them back to the database or removing them, and permanently deleting pollution source nodes, the pollution source is eliminated at its root while maximizing the retention of effective labeled data.

[0044] refer to Figure 2 In another embodiment of the present invention, the cognitive uncertainty of each labeled sample in three modalities is extracted, including: Image data is input into a pre-trained visual perception model. After performing image recognition and image detection on the image data, the visual perception model outputs the first probability distribution of the image data belonging to each category. The first information entropy is calculated based on the first probability distribution and used as the first cognitive uncertainty of the image data. Text data is input into a pre-trained text encoding model. After performing semantic segmentation and semantic classification on the text data, the text encoding model outputs the second probability distribution of the text data belonging to each category. The second information entropy is calculated based on the second probability distribution and used as the second cognitive uncertainty of the text data. The audio data is input into a pre-trained audio coding model. The audio coding model outputs the third probability distribution of the audio data belonging to each category. The third information entropy is calculated based on the third probability distribution and used as the third cognitive uncertainty of the audio data.

[0045] In this embodiment, image data refers to visual information stored in the form of a pixel matrix in the original multimodal dataset. Each pixel contains color and brightness values, and multiple pixels are arranged in rows and columns to form a two-dimensional image matrix. The width, height, and number of channels of this matrix together determine the size and color depth of the image data.

[0046] In this embodiment, the pre-trained visual perception model refers to a deep learning model pre-trained using a convolutional neural network architecture. During the training phase, the model is supervised by a large-scale image classification dataset containing image data and their corresponding category labels. The model parameters are adjusted through the backpropagation algorithm so that the model can extract visual features from the input image and map them to the category probability space. After the model is trained, its parameters remain unchanged. The model takes image data as input and outputs the probability distribution of the image belonging to each category in the preset category set after multiple convolution and pooling operations.

[0047] In this embodiment, the category refers to each tag item in a pre-defined tag set used to classify multimodal data. This tag set is predefined by the operator according to the specific application scenario. For example, in the medical imaging scenario, the category may include tags such as normal, inflammation, and tumor, while in the industrial quality inspection scenario, the category may include tags such as qualified, defect 1, and defect 2.

[0048] In this embodiment, image data is input into a pre-trained visual perception model. After performing image recognition and image detection on the image data, the visual perception model outputs the first probability distribution of the image data belonging to each category. This means that the image data in the original multimodal dataset is passed as input to the pre-trained visual perception model. After receiving the image data, the model extracts visual features such as edges, textures, and shapes from the image through convolutional layers, and then maps the visual features to the response values ​​of each category in the preset category set through fully connected layers. Finally, the response values ​​are converted into probability values ​​through a normalized exponential function. The sum of the probability values ​​of all categories is equal to 1. The set of these probability values ​​is the first probability distribution.

[0049] In this embodiment, calculating the first information entropy based on the first probability distribution means substituting the probability value corresponding to each category in the first probability distribution into the information entropy calculation formula for calculation. The information entropy is calculated by taking the natural logarithm of the probability value of each category and multiplying it by the probability value, taking the opposite of the product of all categories and adding them together to obtain the first information entropy. The first information entropy is used to quantify the uncertainty of the visual perception model in classifying image data.

[0050] In this embodiment, text data refers to language information stored in the form of character sequences in the original multimodal dataset. The character sequence is composed of multiple characters arranged in a specific order. The characters may include Chinese characters, letters, numbers, punctuation marks and other language symbols. The sequence of multiple characters forms language units of different lengths such as words, sentences or paragraphs.

[0051] In this embodiment, the pre-trained text encoding model refers to a deep learning model pre-trained using the Transformer neural network architecture. During the training phase, the model is supervised by a large-scale text classification dataset containing text data and its corresponding category labels. The model parameters are adjusted through the backpropagation algorithm so that the model can extract semantic features from the input text and map them to the category probability space. After the model is trained, its parameters remain unchanged. The model takes text data as input and outputs the probability distribution of the text belonging to each category in the preset category set after processing by a multi-head self-attention mechanism and a feedforward network.

[0052] In this embodiment, text data is input into a pre-trained text encoding model. After performing semantic segmentation and semantic classification on the text data, the text encoding model outputs the second probability distribution of the text data belonging to each category. This means that the text data in the original multimodal dataset is passed as input to the pre-trained text encoding model. After receiving the text data, the model first converts each character into a vector representation, then captures the semantic association between characters at different positions in the text through a multi-head self-attention mechanism, divides the text into segments with independent semantic meanings through semantic segmentation, maps the semantic features of each segment to the response value of each category in the preset category set through semantic classification, and finally converts the response value into a probability value through a normalized exponential function. The sum of the probability values ​​of all categories is equal to 1, and the set of these probability values ​​is the second probability distribution.

[0053] In this embodiment, calculating the second information entropy based on the second probability distribution means substituting the probability value corresponding to each category in the second probability distribution into the information entropy calculation formula for calculation. The information entropy is calculated by taking the natural logarithm of the probability value of each category and multiplying it by the probability value, taking the opposite of the product of all categories and adding them together to obtain the second information entropy. This second information entropy is used to quantify the uncertainty of the text encoding model in classifying text data.

[0054] In this embodiment, audio data refers to sound information stored in the form of waveform sampling sequences in the original multimodal dataset. The waveform sampling sequence consists of multiple sampling points arranged in chronological order. Each sampling point records the amplitude value of the sound signal at that moment. The sampling frequency and the number of sampling bits together determine the temporal resolution and amplitude accuracy of the audio data.

[0055] In this embodiment, the pre-trained audio coding model refers to a deep learning model pre-trained using a hybrid architecture combining convolutional neural networks and recurrent neural networks. During the training phase, the model is supervised by a large-scale audio classification dataset containing audio data and its corresponding category labels. The model parameters are adjusted through the backpropagation algorithm so that the model can extract acoustic features from the input audio and map them to the category probability space. After the model is trained, its parameters remain fixed. The model takes audio data as input, extracts local time-frequency features through convolutional layers, captures temporal dependencies through recurrent neural network layers, and outputs the probability distribution of the audio belonging to each category in the preset category set.

[0056] In this embodiment, the audio data is input into a pre-trained audio coding model. The third probability distribution of the audio data output by the audio coding model refers to the process of passing the audio data from the original multimodal dataset as input to the pre-trained audio coding model. After receiving the audio data, the model first converts the waveform sampling sequence into a time-frequency graph representation, then extracts local acoustic features from the time-frequency graph through a convolutional layer, captures the change pattern of acoustic features over time through a recurrent neural network layer, and finally maps the acoustic features to the response value of each category in the preset category set through a fully connected layer. Then, the response value is converted into a probability value through a normalized exponential function. The sum of the probability values ​​of all categories is equal to 1. The set of these probability values ​​is the third probability distribution.

[0057] In this embodiment, calculating the third information entropy based on the third probability distribution means substituting the probability value corresponding to each category in the third probability distribution into the information entropy calculation formula for calculation. The information entropy is calculated by taking the natural logarithm of the probability value of each category and multiplying it by the probability value, taking the opposite of the product of all categories and adding them together to obtain the third information entropy. This third information entropy is used to quantify the uncertainty of the audio coding model in the classification decision of audio data.

[0058] The beneficial effects of the above technical solution are as follows: cognitive uncertainty is extracted by using pre-trained models for each of the three modalities of image, text and audio respectively, and the degree of certainty of the model’s classification decision for each category is quantified by information entropy value. The separate processing of the three modalities avoids cross-modal feature confusion and provides reliable input data for the accurate calculation of cross-modal propagation attenuation rate.

[0059] In another embodiment of the invention, all labeled samples are grouped by data type, including: The labeled samples with image data type are divided into image labeled sample groups; The labeled samples with text data type are divided into text labeled sample groups; The labeled samples with audio data type are divided into audio labeled sample groups; Based on the data source as the dividing attribute, the labeled samples are divided into three groups: the first source labeled sample group, the second source labeled sample group, and the third source labeled sample group. Based on the label category, the labeled samples are divided into three categories: Category 1, Category 2, and Category 3.

[0060] In this embodiment, dividing the labeled samples into a first-source labeled sample group, a second-source labeled sample group, and a third-source labeled sample group based on the data source means obtaining the source identification information of each labeled sample at the time of collection or generation. This source identification information is used to distinguish the different source channels of the labeled samples. All labeled samples are assigned to the corresponding groups according to their source identification information. Labeled samples with the first source identification information are assigned to the first-source labeled sample group, labeled samples with the second source identification information are assigned to the second-source labeled sample group, and labeled samples with the third source identification information are assigned to the third-source labeled sample group. All labeled samples in each group come from the same collection or generation channel.

[0061] In this embodiment, the label category refers to each label item in a preset label set used to classify multimodal data. This label set is predefined by the operator according to the specific application scenario. The initial label category label output by the multimodal automatic labeling model when performing initial labeling is a label item in this label category set.

[0062] In this embodiment, dividing the labeled samples into a first category labeled sample group, a second category labeled sample group, and a third category labeled sample group based on the labeled category means obtaining the initial labeled category label for each labeled sample and assigning all labeled samples to the corresponding groups according to the initial labeled category label. Labeled samples with the initial labeled category label of the first category are assigned to the first category labeled sample group, labeled samples with the initial labeled category label of the second category are assigned to the second category labeled sample group, and labeled samples with the initial labeled category label of the third category are assigned to the third category labeled sample group. All labeled samples in each group have the same initial labeled category label.

[0063] The beneficial effects of the above technical solution are as follows: the labeled samples are grouped in multiple dimensions by three classification attributes: data type, data source, and labeling category. This ensures that the samples in each group are consistent in terms of modal attributes, source channels, or labeling categories, providing a structured grouping basis for the calculation of the mean within the group and the analysis of pollution distribution. The three grouping attributes are independent of each other and can be flexibly combined and used.

[0064] refer to Figure 3 In another embodiment of the present invention, tracing the uncertainty propagation path along the direction of decreasing cross-modal propagation attenuation rate includes: The group of labeled samples with the highest pollution transmission asymmetry index in the polluted community was used as the reference group, and the group of labeled samples with the highest ratio of image modal cognitive uncertainty to text modal cognitive uncertainty in the reference group was used as the starting point for tracking. For any two labeled samples within the polluted community, the cognitive uncertainty of the two labeled samples in the image modality is extracted. The absolute value of the difference between the cognitive uncertainties of the two labeled samples in the image modality is used as the basic weight. The basic weight is multiplied by the absolute value of the image-text cross-modal propagation attenuation rate of the reference group to obtain the image-text connection edge weight between the two labeled samples. The cognitive uncertainty of the two labeled samples in the text modality is extracted. The absolute value of the difference between the cognitive uncertainties of the two labeled samples in the text modality is used as the basic weight. The basic weight is multiplied by the absolute value of the text-audio cross-modal propagation attenuation rate of the reference group to obtain the text-audio connection edge weight between the two labeled samples. The image-text connection edge weight and the text-audio connection edge weight are added together to obtain the fused connection edge weight between the two labeled samples. The fused connection edge weight is used as the edge weight to establish a directed connection edge between the labeled samples within the polluted community. The direction of the directed connection edge is from the labeled sample with the higher fused connection edge weight to the labeled sample with the lower fused connection edge weight. Starting from the tracking point, it iterates along the direction of the directed connection edge until a labeled sample with a fused connection edge weight lower than the preset fused connection edge weight threshold is reached. All labeled samples passed during the traversal are then arranged sequentially as uncertainty propagation paths.

[0065] In this embodiment, for any two labeled samples within the polluted community, the cognitive uncertainty of the two labeled samples in the image modality is extracted. The absolute value of the difference between the cognitive uncertainties of the two labeled samples in the image modality is used as the basic weight. The basic weight is multiplied by the absolute value of the image-text cross-modal propagation attenuation rate of the reference group to obtain the image-text connection edge weight between the two labeled samples. The cognitive uncertainty of the two labeled samples in the text modality is extracted. The absolute value of the difference between the cognitive uncertainties of the two labeled samples in the text modality is used as the basic weight. The basic weight is multiplied by the absolute value of the text-audio cross-modal propagation attenuation rate of the reference group to obtain the text-audio connection edge weight between the two labeled samples. The image-text connection edge weight and the text-audio connection edge weight are added together to obtain the fused connection edge weight between the two labeled samples.

[0066] In this embodiment, using the weight of the fused connection edge as the edge weight, establishing a directed connection edge between the labeled samples within the polluted community means treating each labeled sample within the polluted community as a node in a directed network. For any two labeled samples within the polluted community, the weight values ​​of the fused connection edge calculated between the two labeled samples are extracted. The labeled sample with the higher weight of the fused connection edge is taken as the starting point, and the labeled sample with the lower weight of the fused connection edge is taken as the ending point. A directed connection edge is established between the starting point node and the ending point node. The direction of the directed connection edge is from the labeled sample with the higher weight of the fused connection edge to the labeled sample with the lower weight of the fused connection edge. The weight value of the fused connection edge at the ending point is taken as the weight value of the directed connection edge. After performing the same operation on all labeled samples in the polluted community pairwise, a complete directed network structure is formed.

[0067] In this embodiment, starting from the tracking starting point, the process iteratively traverses along the direction of the directed connection edges until a labeled sample whose fusion connection edge weight is lower than a preset fusion connection edge weight threshold is reached. This means taking the tracking starting point as the current node, searching for all directed connection edges originating from the current node in the directed network, selecting the directed connection edge with the largest weight value, moving along the direction of that directed connection edge to the next labeled sample node pointed to by that edge, and taking that next labeled sample node as the new current node. The search and move operations are then repeated. When the fusion connection edge weight of the reached labeled sample node is lower than the preset fusion connection edge weight threshold, the traversal stops, forming a continuous node sequence from the tracking starting point to the terminal node. All labeled samples passed during the traversal are arranged sequentially as uncertainty propagation paths.

[0068] In this embodiment, the preset fusion connection edge weight threshold is a value set by the operator in advance according to the actual application scenario and accuracy requirements. This value is used to determine whether the fusion connection edge weight of the labeled sample is low enough to stop the traversal of the propagation path. When the fusion connection edge weight of the labeled sample is lower than the preset fusion connection edge weight threshold, it means that the uncertainty propagation effect at the sample has decayed to a negligible level, and the sample no longer has the value to continue tracking, and the traversal terminates at the sample.

[0069] In this embodiment, arranging all the labeled samples passed during the traversal into an uncertainty propagation path means starting from the tracking starting point, moving along the directed connection edges one by one, and arranging all the labeled sample nodes visited in the order of their visits into a sequence. The first node in this sequence is the tracking starting point, and the last node is the termination node where the weight of the fused connection edge is lower than a preset fused connection edge weight threshold. There is a directed connection edge between any two adjacent nodes in the sequence, pointing from the node with the higher weight of the fused connection edge to the node with the lower weight of the fused connection edge. This node sequence is the uncertainty propagation path.

[0070] The beneficial effects of the above technical solution are as follows: taking the group with the highest pollution propagation asymmetry index as the reference group, taking the sample with the highest image-text attenuation rate in the group as the tracking starting point, multiplying the two attenuation rates to obtain the fused attenuation rate as the edge weight, constructing directed connection edges in the pollution community along the direction of decreasing attenuation rate and iteratively traversing to generate the propagation path, thus realizing the complete recording of the uncertainty propagation process.

[0071] refer to Figure 3 In another embodiment of the present invention, the labeled sample with the largest rate of decay change in the labeled quality hyperspace along the path is identified as the pollution source node, including: Calculate the difference between the fusion connection edge weight of each labeled sample and the fusion connection edge weight of the previous labeled sample on the uncertainty propagation path, divide the difference by the Euclidean distance between two adjacent labeled samples on the uncertainty propagation path in the labeled quality hyperspace, and obtain the decay rate at the corresponding labeled sample position. Compare the decay change rate at all labeled sample locations, and select the labeled sample with the largest decay change rate as the pollution source node.

[0072] In this embodiment, the Euclidean distance between two adjacent labeled samples in the labeled quality hyperspace refers to the straight-line distance between the corresponding coordinate points of the two adjacent labeled samples on the propagation path in the labeled quality hyperspace. Each labeled sample has a three-dimensional coordinate position in the labeled quality hyperspace determined by three coordinate values: a first cognitive uncertainty, a second cognitive uncertainty, and a third cognitive uncertainty. The three-dimensional coordinate values ​​of the previous labeled sample and the next labeled sample on the propagation path are subtracted from each other, and the squares are taken respectively. The three squares are added together and the square root is taken. The obtained value is the Euclidean distance between the two adjacent labeled samples. This Euclidean distance is used as the normalization benchmark of the denominator when calculating the decay rate of change in the subsequent calculation, so as to eliminate the influence of coordinate scale differences on the calculation of the rate of change.

[0073] The beneficial effects of the above technical solution are as follows: by calculating the difference in fusion attenuation rate between adjacent samples on the propagation path and dividing it by the Euclidean distance in hyperspace, the attenuation change rate is obtained. The sample with the largest change rate is located as the pollution source node. The uncertainty at this location is the fastest and is the starting source that leads to the increase of uncertainty in downstream samples. This solves the defect of existing technology that cannot distinguish between pollution source and secondary pollution.

[0074] In another embodiment of the present invention, contaminated samples are separated from the original multimodal dataset to form a dataset to be cleaned, including: Remove the labeled category tags of contaminated samples from the training sample set of the multimodal automatic labeling model; In the cognitive uncertainty propagation network, the edge weights of the connections between contaminated samples and other labeled samples are set to zero. The contaminated samples are marked with isolation tags and stored in an isolation sample library, which is physically separated from the original multimodal dataset.

[0075] In this embodiment, the label of a contaminated sample refers to the initial label of the contaminated sample in the current dataset. This initial label is the label assigned to the sample by the multimodal automatic labeling model when performing initial labeling.

[0076] In this embodiment, the training sample set of the multimodal automatic annotation model refers to the data set used for incremental training or fine-tuning of the multimodal automatic annotation model. Each sample in the training sample set contains multimodal data and its corresponding annotation category label. The multimodal automatic annotation model continuously reads samples from the training sample set to optimize parameters, thereby improving the model's performance on multimodal data annotation tasks.

[0077] In this embodiment, removing the labeling category of contaminated samples from the training sample set of the multimodal automatic labeling model means that after a contaminated sample is labeled, when preparing training data for updating the multimodal automatic labeling model, the contaminated sample and its corresponding initial labeling category will no longer be included in the training sample set, ensuring that the contaminated sample will not be used as valid training data for updating and iterating model parameters.

[0078] In this embodiment, the cognitive uncertainty propagation network refers to a graph structure network constructed with labeled samples as nodes and cross-modal cognitive uncertainty propagation relationships between labeled samples as connecting edges. Each node in the network represents a labeled sample, and the connecting edge between any two nodes indicates that there is a mutual influence relationship of cross-modal cognitive uncertainty between the two labeled samples. The weight value on the connecting edge is determined by the cross-modal propagation attenuation rate between the two labeled samples.

[0079] In this embodiment, in the cognitive uncertainty propagation network, setting the edge weight of the connection edge between the contaminated sample and other labeled samples to zero means finding all connection edges directly connected to the contaminated sample node in the cognitive uncertainty propagation network and modifying the weight values ​​of these connection edges to 0. After modification, the contaminated sample node no longer has a valid connection with any other labeled sample node, and cross-modal cognitive uncertainty cannot propagate to other labeled sample nodes through the contaminated sample node, thereby cutting off the uncertainty propagation path.

[0080] In this embodiment, attaching an isolation mark to a contaminated sample means creating a corresponding mark for each contaminated sample. This mark is used to uniquely identify that the sample is in an isolated state. The mark is stored in association with the sample identifier, so that subsequent data processing can determine that the corresponding sample has been isolated by recognizing the mark and will not be processed in the usual way. Storing the contaminated sample with the attached isolation mark into the isolation sample library means storing the contaminated sample with the attached isolation mark into the isolation sample library as a whole.

[0081] In this embodiment, the isolated sample library refers to a data storage space that is independent of the original multimodal dataset. This storage space is specifically used to store labeled sample data that are marked as contaminated samples. The data in this storage space is not accessed or used in subsequent routine data processing.

[0082] In this embodiment, the physical separation of the isolated sample library and the original multimodal dataset means that the isolated sample library and the original multimodal dataset are stored on different physical storage media or in different storage areas of the same storage media. The two are independent of each other at the storage level. Operations such as reading, writing, and deleting the original multimodal dataset will not affect the data in the isolated sample library, and operations such as reading, writing, and deleting the isolated sample library will not affect the data in the original multimodal dataset.

[0083] The beneficial effects of the above technical solutions are as follows: by removing the labeled category tags of contaminated samples from the training sample set, setting the weight of the connection edges between contaminated samples and other nodes in the network to zero, and storing the contaminated samples in an isolation sample library that is physically separated from the original dataset after adding isolation tags, triple isolation and blocking at the cognitive, network, and physical levels are achieved.

[0084] In another embodiment of the present invention, re-labeling the dataset to be cleaned includes: Read contaminated samples with attached isolation tags from the isolated sample library; The multimodal automatic annotation model is invoked to perform secondary annotation on the contaminated samples, and the secondary annotation results are obtained. Extract the secondary cognitive uncertainty of each contaminated sample from the secondary annotation results; When the secondary cognitive uncertainty of a contaminated sample is lower than the cognitive uncertainty in the initial labeling results, the contaminated sample is marked as a cleaned sample, and the cleaned sample is removed from the isolated sample library and re-integrated into the original multimodal dataset. When the secondary cognitive uncertainty of a contaminated sample is not lower than the cognitive uncertainty in the initial labeling results, the contaminated sample is marked as a persistent contaminated sample, and the persistent contaminated sample is kept in the isolated sample library and a manual review is triggered.

[0085] In this embodiment, reading contaminated samples with added isolation tags from the isolation sample library means that when relabeling contaminated samples, the isolated sample library, an independent storage space, is accessed to extract the contaminated sample data and its associated isolation tag information one by one, and the extracted contaminated samples are used as input data for subsequent relabeling operations.

[0086] In this embodiment, calling the multimodal automatic annotation model to perform secondary annotation on the contaminated samples and obtaining the secondary annotation result means inputting the contaminated samples read from the isolated sample library into the multimodal automatic annotation model. The multimodal automatic annotation model receives the image data, text data and audio data in the contaminated samples, and outputs the probability distribution of the contaminated sample belonging to each category in the preset category set after model calculation. The category with the highest probability value is used as the secondary annotation category label of the contaminated sample. The secondary annotation category label is the secondary annotation result.

[0087] In this embodiment, extracting the secondary cognitive uncertainty of each contaminated sample in the secondary annotation results means that after the multimodal automatic annotation model performs secondary annotation on the contaminated sample, the probability distribution of the contaminated sample in the image modality is obtained and the information entropy of the probability distribution is calculated as the secondary cognitive uncertainty of the image modality; the probability distribution of the contaminated sample in the text modality is obtained and the information entropy of the probability distribution is calculated as the secondary cognitive uncertainty of the text modality; and the probability distribution of the contaminated sample in the audio modality is obtained and the information entropy of the probability distribution is calculated as the secondary cognitive uncertainty of the audio modality.

[0088] In this embodiment, the fact that the secondary cognitive uncertainty of a contaminated sample is lower than the cognitive uncertainty in the initial annotation result means that, for the same contaminated sample, the secondary cognitive uncertainty of the contaminated sample in the image modality is compared with the initial cognitive uncertainty of the contaminated sample in the image modality, the secondary cognitive uncertainty of the contaminated sample in the text modality is compared with the initial cognitive uncertainty of the contaminated sample in the text modality, and the secondary cognitive uncertainty of the contaminated sample in the audio modality is compared with the initial cognitive uncertainty of the contaminated sample in the audio modality. When the secondary cognitive uncertainty in all three modalities is lower than the initial cognitive uncertainty in the corresponding modality, it is confirmed that the secondary cognitive uncertainty of the contaminated sample is lower than the cognitive uncertainty in the initial annotation result.

[0089] In this embodiment, removing the purified sample from the isolated sample library and then re-incorporating it into the original multimodal dataset means that when a contaminated sample is determined to be a purified sample, the data content of the purified sample and its corresponding secondary labeling category label are deleted from the isolated sample library, and the data content of the purified sample and its corresponding secondary labeling category label are stored in the original multimodal dataset, so that the purified sample becomes a component of the original multimodal dataset again.

[0090] In this embodiment, keeping the stubbornly contaminated sample in the isolation sample library and triggering manual review means that when the secondary cognitive uncertainty of the contaminated sample is not lower than the cognitive uncertainty in the initial labeling result, the contaminated sample is not removed from the isolation sample library, but continues to be stored in the isolation sample library. At the same time, a prompt message is generated and sent to the operator. The prompt message contains the sample identifier of the stubbornly contaminated sample and its cognitive uncertainty values ​​in three modes, prompting the operator to perform manual inspection and labeling of the stubbornly contaminated sample.

[0091] The beneficial effects of the above technical solution are as follows: by using the fact that the cognitive uncertainty of the three modalities is lower than the initial value after secondary labeling as the purification judgment standard, the purified samples are removed from the isolation library and re-integrated into the original dataset, while the stubborn contaminated samples are kept in the isolation library and triggered for manual review, thus realizing the diversion and disposal of contaminated samples.

[0092] In another embodiment of the invention, it further includes: Calculate the contamination concentration within each labeled sample group. The contamination concentration within a group is the ratio of the number of contaminated samples in the labeled sample group to the total number of labeled samples in the current labeled sample group. The product of the pollution propagation asymmetry index and the pollution concentration within each labeled sample group is used as the pollution propagation potential of the current labeled sample group. The pollution propagation potential of all labeled sample groups is normalized to obtain the normalized pollution propagation potential of each labeled sample group. The labeled sample groups whose normalized contamination propagation potential energy is higher than the preset potential energy threshold are marked as priority cleaning groups, and the contaminated samples in the priority cleaning groups are re-labeled first.

[0093] In this embodiment, the intra-group contamination concentration of each labeled sample group is calculated. The intra-group contamination concentration is the ratio of the number of contaminated samples in the labeled sample group to the total number of labeled samples in the current labeled sample group. This means that for each labeled sample group, the number of samples that have been marked as contaminated in the group is counted, and the total number of all labeled samples in the group is counted. The number of contaminated samples is divided by the total number of labeled samples, and the resulting ratio is the intra-group contamination concentration of the labeled sample group. This ratio is used to measure the density of contaminated samples in the current labeled sample group.

[0094] In this embodiment, using the product of the pollution propagation asymmetry index and the pollution concentration within each labeled sample group as the pollution propagation potential energy of the current labeled sample group means that for each labeled sample group, the pollution propagation asymmetry index value and the pollution concentration value within the group are obtained, and the two values ​​are multiplied together. The product of the multiplication is the pollution propagation potential energy of the labeled sample group. This pollution propagation potential energy comprehensively reflects the pollution severity of the labeled sample group in two dimensions: the degree of pollution propagation asymmetry and the density of polluted samples.

[0095] In this embodiment, the pollution propagation potential energy of all labeled sample groups is normalized. To obtain the normalized pollution propagation potential energy of each labeled sample group, the maximum value of the pollution propagation potential energy of all labeled sample groups is calculated first, and then the pollution propagation potential energy of each labeled sample group is divided by the maximum value. The ratio obtained is the normalized pollution propagation potential energy of each labeled sample group. After normalization, the range of pollution propagation potential energy values ​​of all labeled sample groups is compressed to between 0 and 1, which facilitates horizontal comparison between different labeled sample groups.

[0096] In this embodiment, the preset potential energy threshold is a value set by the operator in advance according to the actual application scenario and cleaning priority requirements. The value ranges from 0 to 1 and is used to determine whether the pollution propagation potential energy of the labeled sample group has reached the level that requires it to be marked as a priority cleaning group.

[0097] In this embodiment, the labeled sample groups whose normalized contamination propagation potential energy is higher than the preset potential energy threshold are marked as priority cleaning groups. Prioritizing the relabeling of contaminated samples in the priority cleaning group means comparing the normalized contamination propagation potential energy of each labeled sample group with the preset potential energy threshold. When the normalized contamination propagation potential energy of a labeled sample group is higher than the preset potential energy threshold, the labeled sample group is marked as a priority cleaning group. In the subsequent relabeling process, the relabeling is first performed on the contaminated samples in the priority cleaning group. After all the contaminated samples in the priority cleaning group have been processed, the contaminated samples in other labeled sample groups are then processed.

[0098] The beneficial effects of the above technical solution are as follows: the pollution propagation potential energy is obtained by multiplying the pollution concentration within the group by the asymmetry index. After normalization, it is compared with the preset potential energy threshold. Groups with a value higher than the threshold are marked as priority cleaning groups and re-labeled first, thus realizing priority scheduling and cleaning of the most polluted sample groups.

[0099] In another embodiment of the invention, it further includes: In each labeled sample group, the ratio of the pollution propagation asymmetry index to the pollution concentration within the group is calculated to obtain the pollution source confidence of the current labeled sample group; The labeled sample groups with pollution source confidence levels higher than the preset confidence threshold are marked as high-confidence pollution groups; In the high-confidence pollution group, the contaminated sample with the highest total cognitive uncertainty value is marked as a candidate pollution source sample; Calculate the label deviation between the label category labels of the pollution source candidate samples and the label category labels of other polluted samples in the high-confidence pollution group. The cosine distance between the label vector corresponding to the label category labels of the pollution source candidate samples and the mean vector of the label vector corresponding to the label category labels of other polluted samples in the high-confidence pollution group is used as the label deviation. When the label deviation exceeds a preset deviation threshold, the candidate samples of the pollution source will be permanently removed from the original multimodal dataset.

[0100] In this embodiment, in each labeled sample group, the ratio of the pollution propagation asymmetry index to the pollution concentration within the group is calculated. Obtaining the pollution source confidence of the current labeled sample group means that for each labeled sample group, the pollution propagation asymmetry index value and the pollution concentration value within the group are obtained, and the pollution propagation asymmetry index is divided by the pollution concentration within the group. The quotient obtained is the pollution source confidence of the labeled sample group. The pollution source confidence is used to measure the degree of credibility of the existence of a real pollution source within the current labeled sample group. The higher the ratio, the greater the possibility of the existence of a pollution source within the group.

[0101] In this embodiment, the preset confidence threshold is a value set by the operator in advance based on the actual application scenario's requirements for the accuracy of pollution source identification. This value is used to determine whether the confidence level of the pollution source in the labeled sample group reaches a high confidence level. When the confidence level of the pollution source in the labeled sample group is higher than the preset confidence threshold, it means that the group can be identified as a high-confidence pollution group.

[0102] In this embodiment, marking the contaminated sample with the highest total cognitive uncertainty value as a pollution source candidate sample in the high-confidence contamination group means that for each labeled sample group marked as a high-confidence contamination group, the total cognitive uncertainty value of all contaminated samples in that group is obtained, all contaminated samples are sorted from largest to smallest according to the total cognitive uncertainty value, and the contaminated sample with the largest total cognitive uncertainty value is marked as a pollution source candidate sample. This pollution source candidate sample is the sample most likely to become a real pollution source in the high-confidence contamination group.

[0103] In this embodiment, calculating the label deviation between the label category label of the pollution source candidate sample and the label category labels of other contaminated samples in the high-confidence pollution group means that, for the high-confidence pollution group, the label category label of the pollution source candidate sample is obtained, and the label category labels of all other contaminated samples in the high-confidence pollution group are also obtained. The label category label of the pollution source candidate sample is compared with the label category labels of all other contaminated samples one by one, and the degree of difference is calculated. The overall label deviation between the pollution source candidate sample and other samples in the group is obtained by summing up all the degree of difference.

[0104] In this embodiment, the label vector corresponding to the label category label of the pollution source candidate sample refers to the conversion of the label category label of the pollution source candidate sample into a numerical vector form that can be processed by a computer. This conversion method adopts one-hot encoding, which maps each category in the preset category set to a dimension in the vector space. The dimension corresponding to the label category label of the pollution source candidate sample takes a value of 1, and all other dimensions take a value of 0.

[0105] In this embodiment, the label vector corresponding to the label category label of other contaminated samples in the high-confidence contamination group refers to converting the label category label of each contaminated sample in the high-confidence contamination group, excluding the candidate samples of the contamination source, into a numerical vector form that can be processed by a computer. Each label category label is converted into a corresponding label vector using a one-hot encoding method. Each category in the preset category set is mapped to a dimension in the vector space. The dimension corresponding to the label category label of each contaminated sample is 1, and all other dimensions are 0.

[0106] In this embodiment, the cosine distance between the label vector corresponding to the label category label of the pollution source candidate sample and the mean vector of the label vector corresponding to the label category label of other contaminated samples in the high-confidence pollution group is used as the label deviation. This means that the mean vector of the label vectors of all other contaminated samples in the high-confidence pollution group is first calculated, and the mean vector of the label vector of all other contaminated samples in the high-confidence pollution group is taken as the average value of all other contaminated samples in that dimension. Then, the cosine value of the angle between the label vector of the pollution source candidate sample and the mean vector is calculated. The cosine distance is obtained by subtracting the cosine value of the angle from 1. The cosine distance is the label deviation between the label category label of the pollution source candidate sample and the label category label of other contaminated samples in the high-confidence pollution group. The larger the value, the greater the difference between the label of the pollution source candidate sample and the overall label direction of other samples.

[0107] In this embodiment, the preset deviation threshold is a value set by the operator in advance based on the tolerance of label consistency in the actual application scenario. This value is used to determine whether the labeling category label of the pollution source candidate sample deviates from the overall labeling direction of other samples in the group. When the label deviation is higher than the preset deviation threshold, it indicates that the labeling category label of the pollution source candidate sample is significantly different from the labeling category label of other samples in the group.

[0108] In this embodiment, permanently removing a pollution source candidate sample from the original multimodal dataset means that when the label deviation of a pollution source candidate sample exceeds a preset deviation threshold, the complete data content of the pollution source candidate sample is completely deleted from the original multimodal dataset. This deletion operation is irreversible, and the deleted pollution source candidate sample will no longer participate in any subsequent data processing.

[0109] The beneficial effects of the above technical solution are as follows: the confidence level of pollution sources is calculated by the ratio of the asymmetric index to the pollution concentration, high-confidence pollution groups are screened, the label deviation is calculated by the cosine distance of the label vector, and candidate samples with deviation exceeding the threshold are permanently removed from the original dataset, thus achieving accurate identification and removal of high-confidence pollution sources.

[0110] In another embodiment of the invention, it further includes: In a cognitive uncertainty propagation network, calculate the propagation path length from each contaminated sample in a contaminated community to the source node. Based on the propagation path lengths of all contaminated samples, the distribution of propagation path lengths in the contaminated community is constructed. When the skewness coefficient of the propagation path length distribution exceeds the preset skewness threshold, it is determined that there is multi-source pollution in the polluted community, and the pollution source node localization is re-executed within the polluted community until all pollution source nodes are located. All identified pollution source nodes are permanently removed from the original multimodal dataset as pollution sources to be removed.

[0111] In this embodiment, in the cognitive uncertainty propagation network, calculating the propagation path length from each contaminated sample to the pollution source node within the contaminated community means, for each node marked as a contaminated sample within the contaminated community, finding the shortest path from the contaminated sample node to the pollution source node in the cognitive uncertainty propagation network, counting the number of directed connection edges contained in the shortest path, and using the counted number of edges as the propagation path length from the contaminated sample to the pollution source node.

[0112] In this embodiment, constructing the transmission path length distribution of the contaminated community based on the transmission path length of all contaminated samples means obtaining the transmission path length value corresponding to each of the contaminated samples in the contaminated community, classifying and summarizing all contaminated samples according to the transmission path length value, counting the number of contaminated samples corresponding to each transmission path length value, and forming the transmission path length distribution of the contaminated community with the transmission path length value as the horizontal axis and the corresponding number of contaminated samples as the vertical axis.

[0113] In this embodiment, the skewness coefficient of the transmission path length distribution refers to the statistical quantity obtained by performing skewness statistical calculation on the transmission path length distribution of the contaminated community. The skewness coefficient is used to measure the direction and degree of skewness of the transmission path length distribution relative to the normal distribution. When the skewness coefficient is greater than 0, it indicates that the distribution is positively skewed, that is, most of the contaminated samples are concentrated on the side of the shorter transmission path. When the skewness coefficient is less than 0, it indicates that the distribution is negatively skewed, that is, most of the contaminated samples are concentrated on the side of the longer transmission path. When the skewness coefficient is close to 0, it indicates that the distribution is close to symmetrical.

[0114] In this embodiment, the preset skewness threshold is a value set by the operator in advance based on the sensitivity requirements of multi-source pollution determination in the actual application scenario. This value is a positive number greater than 0 and is used to determine whether there is a significant skewness in the distribution of the transmission path length of the polluted community. When the absolute value of the skewness coefficient is higher than the preset skewness threshold, it indicates that there is a significant skewness in the distribution of the transmission path length and that there are multiple independent sources of pollution transmission within the polluted community.

[0115] In this embodiment, the existence of multi-source pollution in a polluted community means that when the absolute value of the skewness coefficient of the propagation path length distribution of the polluted community exceeds the preset skewness threshold, it is determined that the pollution in the polluted community is not caused by a single pollution source node, but rather by multiple independent pollution source nodes. Each pollution source node has uncertainty in its propagation to the surrounding polluted samples, resulting in the propagation path length distribution exhibiting a multi-peak or significantly skewed shape.

[0116] In this embodiment, re-performing the pollution source node localization within the polluted community until all pollution source nodes are located means that after determining that there is multi-source pollution in the polluted community, the already located pollution source nodes and their associated directed connections are removed from the cognitive uncertainty propagation network. The pollution source node localization operation is then re-performed in the remaining sub-network. Newly located pollution source nodes are marked, and the new pollution source nodes are removed from the network and localization is performed again. This cycle is repeated until no new pollution source nodes can be located from the remaining sub-network. All pollution source nodes located during the cycle are then summarized as the total number of pollution source nodes in the polluted community.

[0117] In this embodiment, all the located pollution source nodes are regarded as pollution sources to be removed. According to the disposal method of the pollution sources to be removed in the embodiment, permanent removal from the original multimodal dataset means that all the labeled samples corresponding to the pollution source nodes located in the pollution community are marked as pollution sources to be removed. For each labeled sample marked as a pollution source to be removed, its complete data content is completely deleted from the original multimodal dataset. This deletion operation is irreversible, and the deleted labeled samples will no longer participate in any subsequent data processing.

[0118] The beneficial effects of the above technical solution are as follows: multi-source pollution is determined by the skewness coefficient of the propagation path length distribution. After removing the located pollution source, the localization is re-executed until no new pollution source node can be located. All located pollution source nodes are permanently removed, which solves the defect that the single pollution source localization method cannot handle multi-source mixed pollution.

[0119] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for cleaning and labeling multimodal datasets based on artificial intelligence, characterized in that, include: Obtain the original multimodal dataset containing image, text, and audio data, call the multimodal automatic annotation model to perform initial annotation, and obtain the initial annotation results for each annotated sample; The cognitive uncertainty of each labeled sample in the three modalities is extracted respectively, and the labeling quality hyperspace is constructed with the three cognitive uncertainties as coordinate axes; All labeled samples were grouped by data type, and the within-group mean of each group was calculated for the three cognitive uncertainties. The image-text cross-modal propagation attenuation rate, the text-audio cross-modal propagation attenuation rate, and the pollution propagation asymmetry index were calculated in turn. Groups with a pollution propagation asymmetry index higher than a preset threshold are marked as polluted communities. The image-text cross-modal propagation attenuation rate and the text-audio cross-modal propagation attenuation rate are used as the basis for calculating the connection edge weights during path tracing. The uncertainty propagation path is traced along the direction of decreasing cross-modal propagation attenuation rate. The labeled sample with the largest attenuation change rate in the labeled quality hyperspace on the path is identified as the pollution source node. The labeled samples in the polluted community other than the pollution source node are marked as polluted samples. The pollution source node is marked as a pollution source to be removed. The contaminated samples are separated from the original multimodal dataset to form the dataset to be cleaned. The dataset to be cleaned is relabeled. The contaminated samples whose cognitive uncertainty is reduced after relabeling are returned to the original multimodal dataset. The contaminated samples whose cognitive uncertainty is not reduced after relabeling are removed from the original multimodal dataset. The contaminants to be removed are permanently removed from the original multimodal dataset to obtain the cleaned multimodal labeled dataset.

2. The method for cleaning and labeling multimodal datasets based on artificial intelligence according to claim 1, characterized in that, Extract the cognitive uncertainty of each labeled sample across three modalities, including: Image data is input into a pre-trained visual perception model. After performing image recognition and image detection on the image data, the visual perception model outputs the first probability distribution of the image data belonging to each category. The first information entropy is calculated based on the first probability distribution and used as the first cognitive uncertainty of the image data. Text data is input into a pre-trained text encoding model. After performing semantic segmentation and semantic classification on the text data, the text encoding model outputs the second probability distribution of the text data belonging to each category. The second information entropy is calculated based on the second probability distribution and used as the second cognitive uncertainty of the text data. The audio data is input into a pre-trained audio coding model. The audio coding model outputs the third probability distribution of the audio data belonging to each category. The third information entropy is calculated based on the third probability distribution and used as the third cognitive uncertainty of the audio data.

3. The method for cleaning and labeling multimodal datasets based on artificial intelligence according to claim 1, characterized in that, Group all labeled samples by data type, including: The labeled samples with image data type are divided into image labeled sample groups; The labeled samples with text data type are divided into text labeled sample groups; The labeled samples with audio data type are divided into audio labeled sample groups; Based on the data source as the dividing attribute, the labeled samples are divided into three groups: the first source labeled sample group, the second source labeled sample group, and the third source labeled sample group. Based on the label category, the labeled samples are divided into three categories: Category 1, Category 2, and Category 3.

4. The method for cleaning and labeling multimodal datasets based on artificial intelligence according to claim 1, characterized in that, Tracing the uncertainty propagation path along the direction of decreasing cross-modal propagation attenuation rate includes: The group of labeled samples with the highest pollution transmission asymmetry index in the polluted community was used as the reference group, and the group of labeled samples with the highest ratio of image modal cognitive uncertainty to text modal cognitive uncertainty in the reference group was used as the starting point for tracking. For any two labeled samples within the polluted community, the cognitive uncertainty of the two labeled samples in the image modality is extracted. The absolute value of the difference between the cognitive uncertainties of the two labeled samples in the image modality is used as the basic weight. The basic weight is multiplied by the absolute value of the image-text cross-modal propagation attenuation rate of the reference group to obtain the image-text connection edge weight between the two labeled samples. The cognitive uncertainty of the two labeled samples in the text modality is extracted. The absolute value of the difference between the cognitive uncertainties of the two labeled samples in the text modality is used as the basic weight. The basic weight is multiplied by the absolute value of the text-audio cross-modal propagation attenuation rate of the reference group to obtain the text-audio connection edge weight between the two labeled samples. The image-text connection edge weight and the text-audio connection edge weight are added together to obtain the fused connection edge weight between the two labeled samples. The fused connection edge weight is used as the edge weight to establish a directed connection edge between the labeled samples within the polluted community. The direction of the directed connection edge is from the labeled sample with the higher fused connection edge weight to the labeled sample with the lower fused connection edge weight. Starting from the tracking point, it iterates along the direction of the directed connection edge until a labeled sample with a fused connection edge weight lower than the preset fused connection edge weight threshold is reached. All labeled samples passed during the traversal are then arranged sequentially as uncertainty propagation paths.

5. The method for cleaning and labeling multimodal datasets based on artificial intelligence according to claim 4, characterized in that, The labeled samples with the largest rate of decay change in the labeled quality hyperspace along the path are identified as pollution source nodes, including: Calculate the difference between the fusion connection edge weight of each labeled sample and the fusion connection edge weight of the previous labeled sample on the uncertainty propagation path, divide the difference by the Euclidean distance between two adjacent labeled samples on the uncertainty propagation path in the labeled quality hyperspace, and obtain the decay rate at the corresponding labeled sample position. Compare the decay change rate at all labeled sample locations, and select the labeled sample with the largest decay change rate as the pollution source node.

6. The method for cleaning and labeling multimodal datasets based on artificial intelligence according to claim 1, characterized in that, The contaminated samples are separated from the original multimodal dataset to form the dataset to be cleaned, including: Remove the labeled category tags of contaminated samples from the training sample set of the multimodal automatic labeling model; In the cognitive uncertainty propagation network, the edge weights of the connections between contaminated samples and other labeled samples are set to zero. The contaminated samples are marked with isolation tags and stored in an isolation sample library, which is physically separated from the original multimodal dataset.

7. The method for cleaning and labeling multimodal datasets based on artificial intelligence according to claim 6, characterized in that, Re-labeling is performed on the dataset to be cleaned, including: Read contaminated samples with attached isolation tags from the isolated sample library; The multimodal automatic annotation model is invoked to perform secondary annotation on the contaminated samples, and the secondary annotation results are obtained. Extract the secondary cognitive uncertainty of each contaminated sample from the secondary annotation results; When the secondary cognitive uncertainty of a contaminated sample is lower than the cognitive uncertainty in the initial labeling results, the contaminated sample is marked as a cleaned sample, and the cleaned sample is removed from the isolated sample library and re-integrated into the original multimodal dataset. When the secondary cognitive uncertainty of a contaminated sample is not lower than the cognitive uncertainty in the initial labeling results, the contaminated sample is marked as a persistent contaminated sample, and the persistent contaminated sample is kept in the isolated sample library and a manual review is triggered.

8. The method for cleaning and labeling multimodal datasets based on artificial intelligence according to claim 1, characterized in that, Also includes: Calculate the contamination concentration within each labeled sample group. The contamination concentration within a group is the ratio of the number of contaminated samples in the labeled sample group to the total number of labeled samples in the current labeled sample group. The product of the pollution propagation asymmetry index and the pollution concentration within each labeled sample group is used as the pollution propagation potential of the current labeled sample group. The pollution propagation potential of all labeled sample groups is normalized to obtain the normalized pollution propagation potential of each labeled sample group. The labeled sample groups whose normalized contamination propagation potential energy is higher than the preset potential energy threshold are marked as priority cleaning groups, and the contaminated samples in the priority cleaning groups are re-labeled first.

9. The method for cleaning and labeling multimodal datasets based on artificial intelligence according to claim 8, characterized in that, Also includes: In each labeled sample group, the ratio of the pollution propagation asymmetry index to the pollution concentration within the group is calculated to obtain the pollution source confidence of the current labeled sample group; The labeled sample groups with pollution source confidence levels higher than the preset confidence threshold are marked as high-confidence pollution groups; In the high-confidence pollution group, the contaminated sample with the highest total cognitive uncertainty value is marked as a candidate pollution source sample; Calculate the label deviation between the label category labels of the pollution source candidate samples and the label category labels of other polluted samples in the high-confidence pollution group. The cosine distance between the label vector corresponding to the label category labels of the pollution source candidate samples and the mean vector of the label vector corresponding to the label category labels of other polluted samples in the high-confidence pollution group is used as the label deviation. When the label deviation exceeds a preset deviation threshold, the candidate samples of the pollution source will be permanently removed from the original multimodal dataset.

10. The method for cleaning and labeling multimodal datasets based on artificial intelligence according to claim 1, characterized in that, Also includes: In a cognitive uncertainty propagation network, calculate the propagation path length from each contaminated sample in a contaminated community to the source node. Based on the propagation path lengths of all contaminated samples, the distribution of propagation path lengths in the contaminated community is constructed. When the skewness coefficient of the propagation path length distribution exceeds the preset skewness threshold, it is determined that there is multi-source pollution in the polluted community, and the pollution source node localization is re-executed within the polluted community until all pollution source nodes are located. All identified pollution source nodes are permanently removed from the original multimodal dataset as pollution sources to be removed.

Images