Label revision method, system and equipment for multi-modal data and medium
By preprocessing multimodal data and representing cognitive potential, the problem of label noise in multimodal data is solved, the accuracy and robustness of label revision are improved, data quality is enhanced, and reliable data support is provided for machine learning models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- STATE GRID ECONOMIC TECH RES INST CO LTD
- Filing Date
- 2025-12-12
- Publication Date
- 2026-04-17
AI Technical Summary
Existing data label generation methods are inefficient and inaccurate when dealing with large-scale, dynamically changing multimodal data, and cannot effectively deal with label noise, affecting the accuracy and robustness of machine learning models.
By acquiring and preprocessing multimodal data, calculating the semantic consistency and prediction probability between modalities, generating cognitive potential to determine label confidence, and introducing weight factors for dynamic correction, a cognitive potential representation characterizing label reliability is constructed, thus optimizing the label revision process.
It significantly improves the accuracy and robustness of multimodal data label revision, effectively suppresses noisy labels, enhances the interpretability and consistency of label revision, and provides reliable data support.
Smart Images
Figure CN121880922A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing, and in particular to a method, system, device, and medium for revising labels for multimodal data. Background Technology
[0002] With the rapid increase in internet data volume, data tag management has become increasingly complex. Traditional data tag generation methods (such as manual annotation or rule-based automated algorithms) are inefficient and inaccurate when dealing with large-scale, dynamically changing data. In particular, as the data changes, tags may become outdated or no longer applicable.
[0003] In practical applications, data labels may become noisy (i.e., incorrect or inconsistent labels) due to labeling errors, data inconsistencies, or the subjectivity of manual labeling. These noisy labels can significantly affect the training performance of machine learning models, reducing their accuracy and robustness. Existing label revision techniques are mainly based on unimodal processing and cannot handle multimodal information, resulting in low data quality.
[0004] Therefore, how to effectively label data and provide reliable data support for the operation and maintenance process has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] This invention provides a method, system, device, and medium for revising labels for multimodal data, addressing how to construct cognitive potential energy characterizing label reliability to revise labels and improve the reliability of label revision.
[0006] To address the aforementioned technical problems, embodiments of the present invention provide a label revision method for multimodal data, comprising: Multimodal data labeled with initial tags is acquired and preprocessed to obtain a sample modal feature set; the sample modal feature set includes data from at least two modalities for each sample; The semantic consistency between modalities within each sample is calculated based on the sample modal feature set, and the sample modal feature set is input into a preset classification model to output the predicted probability of each sample belonging to the target category. Based on the semantic consistency and the prediction probability, the cognitive potential of each sample corresponding to each modality is obtained, and the label confidence of each sample in each modality is determined by the cognitive potential. A weighting factor is introduced to dynamically correct the cognitive potential of samples whose label confidence meets the preset conditions, so as to obtain a comprehensive cognitive potential representation of each sample. The initial label is revised using the comprehensive cognitive potential representation.
[0007] Furthermore, the step of calculating the semantic consistency between modalities within each sample based on the sample modal feature set includes: A coherence potential function is constructed based on the sample modal feature set, and the coherence potential function is used to calculate the semantic distance between intra-sample modes in the sample modal feature set; The semantic distance is mapped to the intra-sample inter-modal consistency score; The consistency scores are aggregated and output, and the semantic consistency degree is determined based on the output results.
[0008] Furthermore, the step of obtaining the cognitive potential of each sample corresponding to each modality based on the semantic consistency degree and the prediction probability includes: The prediction uncertainty of each sample corresponding to the current mode is determined by the prediction probability, and the prediction uncertainty is used as the mode entropy to generate a cognitive potential mapping matrix; The semantic consistency level is input into the cognitive potential mapping matrix to obtain the cognitive potential.
[0009] Furthermore, the introduction of weighting factors to dynamically adjust the cognitive potential corresponding to samples whose label confidence meets preset conditions includes: Samples with a label confidence level lower than a preset threshold are removed to obtain highly reliable samples; The high-confidence samples are input into a preset modulation network for weight learning to obtain the dynamic weights output by the modulation network. The cognitive potential is dynamically corrected using the dynamic weights.
[0010] Furthermore, the step of acquiring multimodal data labeled with initial labels and preprocessing it to obtain a sample modality feature set includes: The multimodal data is preprocessed, including noise removal, standardization and modality alignment, to obtain initial modal features. In the modality alignment stage, different modalities are mapped to a shared semantic space and a loss function is constructed for alignment optimization. When a mode is detected to be missing in the initial mode features, the initial mode features are input into a pre-trained distillation network to generate missing mode features; With the goal of minimizing KL divergence loss, the consistency between the modes of the missing modal features is optimized to obtain the sample modal feature set.
[0011] Furthermore, after revising the initial label using the comprehensive cognitive potential representation, the method further includes: The revised training set, which carries the revised labels, obtained by revising the initial labels, is input into the classification model; Calculate the loss between the output of the classification model and the revised label, and update the parameters of the classification model to obtain the trained classification model; A validation set is introduced into the trained classification model for performance validation.
[0012] Furthermore, the method also includes: Based on the performance verification results, a composite objective function based on a meta-learning architecture is constructed. The composite objective function is driven to update the hyperparameters in the comprehensive cognitive potential representation using a gradient descent mechanism.
[0013] Another embodiment of the present invention provides a label revision system for multimodal data, comprising: The sample set construction module is used to acquire multimodal data labeled with initial labels and preprocess it to obtain a sample modality feature set; the sample modality feature set includes data from at least two modalities for each sample; The consistency analysis module is used to calculate the semantic consistency between modalities within each sample based on the sample modal feature set, and input the sample modal feature set into a preset classification model to output the predicted probability of each sample belonging to the target category; The potential energy calculation module is used to obtain the cognitive potential energy of each sample corresponding to each modality based on the semantic consistency degree and the prediction probability, and to determine the label confidence of each sample in each modality based on the cognitive potential energy. The potential energy optimization module is used to introduce weighting factors to dynamically correct the cognitive potential energy corresponding to samples whose label confidence meets preset conditions, so as to obtain a comprehensive cognitive potential energy representation for each sample. The label revision module is used to revise the initial label using the comprehensive cognitive potential representation.
[0014] Furthermore, the consistency analysis module is specifically used for: A coherence potential function is constructed based on the sample modal feature set, and the coherence potential function is used to calculate the semantic distance between intra-sample modes in the sample modal feature set; The semantic distance is mapped to the intra-sample inter-modal consistency score; The consistency scores are aggregated and output, and the semantic consistency degree is determined based on the output results.
[0015] Furthermore, the potential energy calculation module is specifically used for: The prediction uncertainty of each sample corresponding to the current mode is determined by the prediction probability, and the prediction uncertainty is used as the mode entropy to generate a cognitive potential mapping matrix; The semantic consistency level is input into the cognitive potential mapping matrix to obtain the cognitive potential.
[0016] Furthermore, the potential energy optimization module is specifically used for: Samples with a label confidence level lower than a preset threshold are removed to obtain highly reliable samples; The high-confidence samples are input into a preset modulation network for weight learning to obtain the dynamic weights output by the modulation network. The cognitive potential is dynamically corrected using the dynamic weights.
[0017] Furthermore, the sample set construction module is specifically used for: The multimodal data is preprocessed, including noise removal, standardization and modality alignment, to obtain initial modal features. In the modality alignment stage, different modalities are mapped to a shared semantic space and a loss function is constructed for alignment optimization. When a mode is detected to be missing in the initial mode features, the initial mode features are input into a pre-trained distillation network to generate missing mode features; With the goal of minimizing KL divergence loss, the consistency between the modes of the missing modal features is optimized to obtain the sample modal feature set.
[0018] Furthermore, after revising the initial label using the comprehensive cognitive potential representation, the method further includes: The revised training set, which carries the revised labels, obtained by revising the initial labels, is input into the classification model; Calculate the loss between the output of the classification model and the revised label, and update the parameters of the classification model to obtain the trained classification model; A validation set is introduced into the trained classification model for performance validation.
[0019] Furthermore, the system also includes a meta-learning module, specifically used for: Based on the performance verification results, a composite objective function based on a meta-learning architecture is constructed. The composite objective function is driven to update the hyperparameters in the comprehensive cognitive potential representation using a gradient descent mechanism.
[0020] Another embodiment of the present invention provides a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the label revision method for multimodal data as described above.
[0021] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the tag revision method for multimodal data as described above.
[0022] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following: This invention obtains aligned sample modal feature sets by acquiring and preprocessing multimodal data, providing high-quality input for subsequent analysis. It then calculates the semantic consistency between different modalities and the class prediction probability of each sample. By coordinating the class prediction probability and semantic consistency, it generates cognitive potential energy reflecting the label credibility under the current modality, thereby adaptively correcting the labels. This accurately identifies and quantifies potential noise labels, significantly improving the accuracy and robustness of noise detection. Furthermore, it introduces weighting factors to optimize the cognitive potential energy, improving its interpretability for cross-modal label confidence, effectively suppressing interference from low-quality modalities, and enhancing the consistency and reliability of multimodal label revision in dynamic environments. Attached Figure Description
[0023] Figure 1 This is a schematic flowchart of a label revision method for multimodal data in one embodiment of the present invention; Figure 2 This is a schematic diagram of the tag revision system structure for multimodal data in one embodiment of the present invention; Figure 3 This is a structural block diagram of a preferred embodiment of a computer device provided by the present invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0025] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0026] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.
[0027] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.
[0028] One embodiment of the present invention provides a tag revision method for multimodal data, applicable to various power grid fields and business scenarios, such as document management, knowledge management, and data analysis. For details, please refer to... Figure 1 , Figure 1 The diagram shown is a flowchart of a label revision method for multimodal data according to one embodiment of the present invention, including the following steps: S1. Obtain multimodal data labeled with initial labels and preprocess it to obtain the sample modal feature set.
[0029] Multimodal data can include data from various modalities such as images, text, and sensors. This embodiment preferentially uses the HatefulMemes Challenge (HMC) dataset as the primary experimental dataset. This dataset is a typical image-text bimodal dataset with approximately 10,000 samples. Each sample contains an image modality (Memes) and a text modality (text descriptions within the image). Each sample corresponds to a binary label ("Hateful" or "Not Hateful"). Due to the subjectivity of manual annotation, the initial labels may contain potential noise, therefore preprocessing such as timestamp alignment, noise removal, modality alignment, and normalization is required.
[0030] The following formula in this embodiment represents multimodal data: Where i represents the i-th sample in dataset D, Represents the set of all modal features of the i-th sample. and the corresponding initial label , Indicates the number of samples.
[0031] Modal feature set of each sample Specifically, it is expressed as follows: in, Let M represent the set of all modal features for the i-th sample, and m represent the specific modality type. In this example, M=2, representing the image modality and the text modality, respectively.
[0032] Then, timestamp alignment and modality matching are first performed on dataset D. For the text description and image object of each sample, this embodiment uses the unique sample identifier (sample_id) as an index, and achieves modality-level alignment through hash verification and timestamp synchronization to prevent sample mismatch. If the data comes from multiple sources (such as social media or visual crawling platforms), dynamic interpolation will be performed based on the collection timestamp sequence to ensure that each modality record corresponds on a unified timeline, thereby guaranteeing the temporal consistency of modality pairing. Thus, timestamp-aligned multimodal data is obtained.
[0033] Next, noise detection and removal are performed. In this embodiment, a two-level strategy is adopted: the first level is outlier detection based on statistical features, which calculates Mahalanobis distance. The method for identifying abnormal samples is as follows: In the formula, x represents the feature vector of the sample to be detected. This represents the sample mean vector.
[0034] The second-level self-supervised denoising mechanism based on deep features uses an autoencoder to reconstruct the error. As a noise confidence metric, samples with reconstruction errors higher than the global 95th percentile are identified as noise and removed.
[0035] After noise removal, the co-encoder performs data encoding. For image modalities, this embodiment performs pixel normalization and multi-scale enhancement operations. Preferably, the image is first scaled to a uniform resolution of 224×224, and then channel normalization is performed: the pixel value of each channel is normalized by mean-variance, calculated as follows: in, and These represent the channel mean and standard deviation, respectively, and I represents the features of the original image modality. These are the image modal features after normalization. To enhance the robustness of subsequent applications such as classification models, random flipping, random cropping, and color dithering strategies are used for enhanced representation.
[0036] For the text modality, this embodiment performs word segmentation, sub-word embedding, and semantic standardization processes.
[0037] First, the SentencePiece word segmenter is preferably used to perform sub-word segmentation on the text and remove low-frequency words and special symbols; Then, a pre-trained BERT model is used to map each token to an embedding vector. Then, it is mapped to a shared dimension d=512 through a linear transformation layer to ensure consistency with the image modal feature dimension.
[0038] To eliminate the impact of text length differences, a self-attention weighted average is further performed at the sentence level, as shown in the following formula: in, For text modal features, Here, is the attention weight, used to capture the semantic contribution of key information words; t is the index of the current word; j is the traversal index, used to... The summation is performed in the denominator; q is the learnable query vector. This is the transpose of the query vector q.
[0039] Finally, the image and text modal features are standardized and semantically centered to ensure consistent distribution. Specifically, Z-score standardization is preferably used to zero out the mean and normalize the variance of different modal embeddings, making the modal features comparable on the same distribution scale. Semantic centering preferably uses PCA projection to remove principal component drift, ensuring that image and text embeddings have the same semantic distribution basis vectors in a shared space. After this stage of processing, a unified set of standardized multimodal features is obtained, laying the foundation for subsequent self-correcting cognitive potential learning.
[0040] In some embodiments of the present invention, to ensure the alignment effect of mapping different modalities to the shared semantic space during the modality alignment stage, a loss function is further constructed for alignment optimization, preferably using InfoNCE loss. Optimization, as shown below: In the formula, Represents the natural exponential function; Represents the similarity function; This represents the modality 1 feature of the current sample i; This represents the modality 2 feature of the current sample i; The modality 2 feature of the summed sample j is used as a negative sample for contrastive learning; This represents the temperature coefficient, used to control the smoothness of the similarity distribution.
[0041] The loss optimization process first maps features from different modalities to a shared semantic space using a projection network. The goal of InfoNCE loss is to maximize the similarity between different modal features of a sample, while minimizing the similarity between a modal feature of that sample and the different modal features of all other samples. This allows the model to learn a semantic space where different modal representations belonging to the same sample are close to each other, while modal representations belonging to different samples are far apart, thus achieving semantic alignment across modal features and making feature representations modally invariant.
[0042] Through the preprocessing of multimodal data described above, initial modal features are obtained. Then, the missing modal features of these initial features are detected. When a missing modal feature is detected, the initial modal features are further input into a pre-trained distillation network to generate missing modal features. This distillation network consists of three lightweight fully connected (FC) layers, each followed by a LayerNorm layer to prevent gradient explosion. The output dimension of the last layer is consistent with the original feature dimension of the target modality. The specific structure is as follows: in, This represents the predicted mode for the current sample i; For input distillation network The initial modal characteristics; m is the target modal type; Modal features are input to the distillation network; The parameters of the distillation network include all weight matrices W and bias vectors b; , , The weight matrix of the distillation network consists of three lightweight fully connected layers; This is the bias vector for the three fully connected layers; This is the activation function used after the fully connected layer.
[0043] To maintain cross-modal consistency, this embodiment uses KL divergence loss. The objective is to minimize the consistency between modes with missing modal features, specifically as follows: in, This is the Kullback–Leibler divergence term, used to constrain the consistency of the two distributions; The probability distribution output by the teacher network; The probability distribution of the network output for students; Let m be the true features of modality m of sample i in the shared semantic space; the L2 term is used to ensure the numerical stability of the embedding; This is the balance coefficient.
[0044] KL divergence optimization ensures the continuity and robustness of potential energy assessment and label revision even when modes are missing, thus obtaining the required sample modal feature set.
[0045] S2. Calculate the semantic consistency between modalities within each sample based on the sample modal feature set, and input the sample modal feature set into the preset classification model to output the predicted probability of each sample belonging to the target category.
[0046] This embodiment constructs a coherence potential function based on the sample modal feature set to quantify the consistency across modalities. Specifically, the coherence potential function is defined as follows: in, Let be the cross-modal coherence potential of the i-th sample; The modal energy scaling factor, σ, is adaptively determined by the data distribution, with an initial value... σ is the square root of the average Euclidean distance between modes of all samples, and σ is a learnable parameter used to minimize the potential stability loss during training; Z is the normalization term for the number of mode pairs; a is the label confidence of the current sample; b is the preset label confidence threshold.
[0047] Based on the above formula, it can be understood that in the process of using the coherence potential function to quantify the semantic consistency between modes, the semantic distance between modes within each sample in the sample modal feature set is calculated, i.e., the Euclidean distance, by driving the coherence potential function: Then, the semantic distance is mapped to the consistency score between modalities within each sample using the exponential term exp(). The consistency scores are aggregated and output, and the degree of semantic consistency is determined based on the output. That is, if the semantic distance between two modalities is small, the exponential term is close to 1, indicating that the modalities have high semantic consistency.
[0048] Furthermore, in this embodiment, the sample modal feature set is input into a preset classification model to analyze the first... The probability that a sample is predicted to be of class l Specifically, it is expressed as follows: in, Represents a classifier. This is the modal feature vector of the i-th sample in the modal feature set output by the encoder.
[0049] S3. Based on the semantic consistency and prediction probability, obtain the cognitive potential of each sample for each modality, and use the cognitive potential to determine the label confidence of each sample in each modality.
[0050] It should be understood that the prediction probability can characterize the degree of prediction uncertainty for each sample corresponding to the current modality. Therefore, in this embodiment, the degree of prediction uncertainty is used as the modality entropy, and together with the previously generated coherent potential function used to quantify the degree of semantic consistency between modalities, a cognitive potential mapping matrix is generated to obtain the cognitive potential, which is specifically expressed by the following formula: in, To understand potential energy, A higher value indicates higher label reliability in that modality; The confidence score output by the model is directly derived from the forward inference result of the classification model on the m-th modal input; The modal entropy term is calculated from the predicted probability distribution using the entropy function. It is used to measure the uncertainty of the prediction of this modality. Its value increases as the probability distribution becomes more uniform and decreases as the probability distribution becomes more concentrated. Therefore, the higher the prediction probability and the stronger the confidence, the lower the corresponding modal entropy. The three terms are related through the concentration of the probability distribution and together determine the quantification result of the cognitive potential under this modality. This is an entropy adjustment coefficient to prevent the model from overconfident. This is the balance factor between confidence and coherence potential.
[0051] In this embodiment, the cognitive potential is represented by a mapping matrix, namely the cognitive potential mapping matrix, which is expressed as: Each row of the matrix corresponds to the potential energy of a sample in M modes, which can be used to quantify the reliability of each sample label, thereby effectively detecting potential noise labels.
[0052] S4~S5: Introduce weighting factors to dynamically correct the cognitive potential of samples whose label confidence meets the preset conditions, obtain the comprehensive cognitive potential representation of each sample, and use the comprehensive cognitive potential representation to revise the initial label.
[0053] First, a preliminary sample screening is performed, removing samples with label confidence levels below a preset threshold to obtain high-confidence samples. For example, a threshold is used... The potential energy is screened, and low potential energy samples are removed. The threshold can be set as the 30th percentile of the potential energy distribution of all samples.
[0054] High-confidence samples are input into a pre-defined modulation network. By performing weight learning, the dynamic weights of the modulation network output are obtained. , means as follows: in, It is a two-layer multilayer perceptron (MLP) network, with each layer followed by ReLU activation and LayerNorm, and softmax to ensure weight normalization.
[0055] Finally, using the aforementioned dynamic weights Cognitive potential Dynamic corrections are made to obtain a comprehensive cognitive potential energy representation. Specifically: in, The consistency coefficient, This is the magnification factor.
[0056] This is used to revise the initial labels, that is, to prune the labels by class. Specifically, this can be done by selecting the bottom 20% of low-potential samples in each category and removing them directly or by manual review. Finally, a revised training set with revised labels is obtained.
[0057] In some embodiments of the present invention, an inner and outer layer dynamic parameter update mechanism is designed. Specifically, in the inner layer update, the revised training set is input into the classification model, the loss between the output result of the classification model and the revised label is calculated, and the classification model parameters, including the weights of the classification model, are updated iteratively. In some embodiments, the encoder parameters / weights are updated simultaneously to obtain the trained classification model.
[0058] A validation set is introduced into the trained classification model for performance verification. In this embodiment, the cumulative error rate (CER) is preferred as the metric to determine the classification accuracy of the labeled samples. The specific definition of this metric is as follows: Among them, if The value is 1 if the value is 1, otherwise the value is 0. This refers to the final revision label.
[0059] Based on the performance verification results, a composite objective function based on a meta-learning architecture is constructed. In this embodiment, it can be a reward function, specifically expressed as: in, For a clean validation set Improved accuracy The final verification error rate for samples judged as noise; It is the coherent potential function; This is for calibration error; For calculating cost regularization; Weights for generalization performance; Adjust the weights for noise; Modal consistency weights; For calibration error weights; This is used to calculate the cost penalty factor.
[0060] Specifically, the composite objective function covers multiple metrics, including validation set accuracy, false positive rate for noisy samples, semantic consistency of the coherence potential function, calibration error, and computational cost. It aims to comprehensively calculate the gradient of R with respect to each hyperparameter to update the hyperparameters. If R improves for three consecutive iterations, the hyperparameters are fixed; otherwise, they are rolled back to the previous state.
[0061] In the outer layer update, the composite objective function R is driven by gradient descent to update the hyperparameters in the comprehensive cognitive potential representation. These hyperparameters specifically include: , as well as Hyperparameters were updated. The comprehensive cognitive potential was recalculated using the updated hyperparameters, and the labels were revised again.
[0062] For example, this revised labeled multimodal dataset can be used to train prediction / diagnostic models / large models for power systems, enabling accurate detection and effective operation and maintenance of power system faults.
[0063] In summary, this invention quantifies the cross-modal semantic consistency of samples in a multimodal dataset by designing a coherence potential function, and calculates the predicted probability of a sample belonging to the target category in conjunction with a classification model. Then, it constructs cognitive potential using the two indicators of semantic consistency and predicted probability to characterize the credibility of sample labels across different modalities, effectively suppressing interference from low-quality modalities. Simultaneously, it designs an objective function for revision performance to optimize parameters, thereby enhancing the interpretability and accuracy of label revision and providing reliable data support for subsequent power operation and maintenance.
[0064] One embodiment of the present invention provides a label revision system for multimodal data. For details, please refer to [link / reference]. Figure 2 , Figure 2 The diagram shown illustrates the structure of a tag revision system for multimodal data in one embodiment of the present invention, including: The sample set construction module M1 is used to acquire multimodal data labeled with initial labels and preprocess it to obtain a sample modality feature set; the sample modality feature set includes data from at least two modalities for each sample; The consistency analysis module M2 is used to calculate the semantic consistency between modalities within each sample based on the sample modal feature set, and input the sample modal feature set into a preset classification model to output the predicted probability of each sample belonging to the target category; The potential energy calculation module M3 is used to obtain the cognitive potential energy of each sample corresponding to each modality based on the semantic consistency degree and the prediction probability, and to determine the label confidence of each sample in each modality based on the cognitive potential energy. The potential energy optimization module M4 is used to introduce weighting factors to dynamically correct the cognitive potential energy corresponding to the samples whose label confidence meets the preset conditions, so as to obtain the comprehensive cognitive potential energy representation of each sample. The label revision module M5 is used to revise the initial label using the comprehensive cognitive potential representation.
[0065] This embodiment preprocesses the data through a sample set construction module. Specifically, the multimodal data undergoes preprocessing including noise removal, standardization, and modality alignment to obtain initial modality features. In the modality alignment stage, different modalities are mapped to a shared semantic space, and a loss function is constructed for alignment optimization. When a modality is detected to be missing from the initial modality features, the initial modality features are input into a pre-trained distillation network to generate missing modality features. With the goal of minimizing the KL divergence loss, the consistency between the modalities of the missing modality features is optimized to obtain the sample modality feature set.
[0066] In this embodiment, semantic consistency analysis and probability prediction between modalities are performed through the consistency analysis module M2. Specifically, a coherence potential function is constructed based on the sample modal feature set, and the coherence potential function is used to calculate the semantic distance between modalities within each sample in the sample modal feature set. The semantic distance is mapped to the consistency score between modalities within each sample. The consistency scores are aggregated and output, and the degree of semantic consistency is determined based on the output results.
[0067] Furthermore, cognitive potential is constructed and label confidence is analyzed through the potential energy calculation module M3. The prediction uncertainty of each sample corresponding to the current mode is determined by the prediction probability, and the prediction uncertainty is used as the mode entropy to generate a cognitive potential mapping matrix. The semantic consistency is input into the cognitive potential mapping matrix to obtain the cognitive potential.
[0068] Furthermore, in this embodiment, the potential energy optimization module M4 is used to remove samples whose label confidence is lower than a preset threshold to obtain high-confidence samples. Then, the high-confidence samples are input into a preset modulation network for weight learning to obtain the dynamic weights output by the modulation network; the cognitive potential energy is dynamically corrected using the dynamic weights.
[0069] In this embodiment, after revising the initial label using the comprehensive cognitive potential representation, the method further includes: The revised training set, which carries the revised labels, is obtained by revising the initial labels and input into the classification model; the loss between the output of the classification model and the revised labels is calculated to update the classification model parameters and obtain the trained classification model; a validation set is introduced into the trained classification model for performance verification.
[0070] In some embodiments of the present invention, a meta-learning module M6 is also included, specifically used for: constructing a composite objective function based on the meta-learning architecture according to the performance verification results; and driving the composite objective function to update the hyperparameters in the comprehensive cognitive potential representation using the gradient descent mechanism.
[0071] like Figure 3 As shown, this embodiment of the invention also provides a computer device. Figure 3 This is a structural block diagram of a preferred embodiment of a computer device provided by the present invention. The computer device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the method described above.
[0072] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.
[0073] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor. The processor is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.
[0074] The memory mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., while the data storage area can store related data, etc. Furthermore, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard drive, a SmartMedia Card (SMC), a Secure Digital (SD) card, and a Flash Card, or other volatile solid-state storage devices.
[0075] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 3 The structural block diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or use different components. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0076] Accordingly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the steps in the method of the above embodiments, for example... Figure 1 Steps S1 to S5 as described above.
[0077] The technical features and effects of the label revision system for multimodal data proposed in this embodiment of the invention are the same as those of the label revision method for multimodal data proposed in this embodiment of the invention, and will not be repeated here.
[0078] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.
Claims
1. A label revision method for multi-modal data, characterized by, include: Obtain multimodal data labeled with initial labels and preprocess it to obtain the sample modal feature set; The sample modal feature set includes data for at least two modalities for each sample; The semantic consistency between modalities within each sample is calculated based on the sample modal feature set, and the sample modal feature set is input into a preset classification model to output the predicted probability of each sample belonging to the target category. Based on the semantic consistency and the prediction probability, the cognitive potential of each sample corresponding to each modality is obtained, and the label confidence of each sample in each modality is determined by the cognitive potential. A weighting factor is introduced to dynamically correct the cognitive potential of samples whose label confidence meets the preset conditions, so as to obtain a comprehensive cognitive potential representation of each sample. The initial label is revised using the comprehensive cognitive potential representation.
2. The multi-modal data oriented label revision method of claim 1, wherein, The step of calculating the semantic consistency between modalities within each sample based on the sample modal feature set includes: A coherence potential function is constructed based on the sample modal feature set, and the coherence potential function is used to calculate the semantic distance between modes within each sample in the sample modal feature set; The semantic distance is mapped to the intra-sample inter-modal consistency score; The consistency scores are aggregated and output, and the semantic consistency degree is determined based on the output results.
3. The multi-modal data oriented label revision method of claim 1, wherein, The step of obtaining the cognitive potential of each sample corresponding to each modality based on the semantic consistency degree and the prediction probability includes: The prediction uncertainty of each sample corresponding to the current mode is determined by the prediction probability, and the prediction uncertainty is used as the mode entropy to generate a cognitive potential mapping matrix; The semantic consistency level is input into the cognitive potential mapping matrix to obtain the cognitive potential.
4. The multi-modal data oriented label revision method of claim 1, wherein, The process of introducing weighting factors to dynamically adjust the cognitive potential corresponding to samples whose label confidence meets preset conditions includes: Samples with a label confidence level lower than a preset threshold are removed to obtain highly reliable samples; The high-confidence samples are input into a preset modulation network for weight learning to obtain the dynamic weights output by the modulation network. The cognitive potential is dynamically corrected using the dynamic weights.
5. The multi-modal data oriented label revision method of claim 1, wherein, The process of acquiring multimodal data labeled with initial tags and preprocessing it to obtain a sample modality feature set includes: The multimodal data is preprocessed, including noise removal, standardization and modality alignment, to obtain initial modal features. In the modality alignment stage, different modalities are mapped to a shared semantic space and a loss function is constructed for alignment optimization. When a mode is detected to be missing in the initial mode features, the initial mode features are input into a pre-trained distillation network to generate missing mode features; With the goal of minimizing KL divergence loss, the consistency between the modes of the missing modal features is optimized to obtain the sample modal feature set.
6. The label revision method for multimodal data as described in claim 1, characterized in that, After revising the initial label using the comprehensive cognitive potential representation, the method further includes: The revised training set, which carries the revised labels, obtained by revising the initial labels, is input into the classification model; Calculate the loss between the output of the classification model and the revised label, and update the parameters of the classification model to obtain the trained classification model; A validation set is introduced into the trained classification model for performance validation.
7. The label revision method for multimodal data as described in claim 6, characterized in that, The method further includes: Based on the performance verification results, a composite objective function based on a meta-learning architecture is constructed. The composite objective function is driven to update the hyperparameters in the comprehensive cognitive potential representation using a gradient descent mechanism.
8. A label revision system for multimodal data, characterized in that, include: The sample set construction module is used to acquire multimodal data labeled with initial labels and preprocess it to obtain a sample modality feature set; the sample modality feature set includes data from at least two modalities for each sample; The consistency analysis module is used to calculate the semantic consistency between modalities within each sample based on the sample modal feature set, and input the sample modal feature set into a preset classification model to output the predicted probability of each sample belonging to the target category; The potential energy calculation module is used to obtain the cognitive potential energy of each sample corresponding to each modality based on the semantic consistency degree and the prediction probability, and to determine the label confidence of each sample in each modality based on the cognitive potential energy. The potential energy optimization module is used to introduce weighting factors to dynamically correct the cognitive potential energy corresponding to samples whose label confidence meets preset conditions, so as to obtain a comprehensive cognitive potential energy representation for each sample. The label revision module is used to revise the initial label using the comprehensive cognitive potential representation.
9. The label revision system for multimodal data as described in claim 8, characterized in that, The consistency analysis module is specifically used for: A coherence potential function is constructed based on the sample modal feature set, and the coherence potential function is used to calculate the semantic distance between modes within each sample in the sample modal feature set; The semantic distance is mapped to the intra-sample inter-modal consistency score; The consistency scores are aggregated and output, and the semantic consistency degree is determined based on the output results.
10. The label revision system for multimodal data as described in claim 8, characterized in that, The potential energy calculation module is specifically used for: The prediction uncertainty of each sample corresponding to the current mode is determined by the prediction probability, and the prediction uncertainty is used as the mode entropy to generate a cognitive potential mapping matrix; The semantic consistency level is input into the cognitive potential mapping matrix to obtain the cognitive potential.
11. The label revision system for multimodal data as described in claim 8, characterized in that, The potential energy optimization module is specifically used for: Samples with a label confidence level lower than a preset threshold are removed to obtain highly reliable samples; The high-confidence samples are input into a preset modulation network for weight learning to obtain the dynamic weights output by the modulation network. The cognitive potential is dynamically corrected using the dynamic weights.
12. The label revision system for multimodal data as described in claim 8, characterized in that, The sample set construction module is specifically used for: The multimodal data is preprocessed, including noise removal, standardization and modality alignment, to obtain initial modal features. In the modality alignment stage, different modalities are mapped to a shared semantic space and a loss function is constructed for alignment optimization. When a mode is detected to be missing in the initial mode features, the initial mode features are input into a pre-trained distillation network to generate missing mode features; With the goal of minimizing KL divergence loss, the consistency between the modes of the missing modal features is optimized to obtain the sample modal feature set.
13. The label revision system for multimodal data as described in claim 8, characterized in that, After revising the initial label using the comprehensive cognitive potential representation, the method further includes: The revised training set, which carries the revised labels, obtained by revising the initial labels, is input into the classification model; Calculate the loss between the output of the classification model and the revised label, and update the parameters of the classification model to obtain the trained classification model; A validation set is introduced into the trained classification model for performance validation.
14. The label revision system for multimodal data as described in claim 13, characterized in that, The system also includes a meta-learning module, specifically used for: Based on the performance verification results, a composite objective function based on a meta-learning architecture is constructed. The composite objective function is driven to update the hyperparameters in the comprehensive cognitive potential representation using a gradient descent mechanism.
15. A computer device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the label revision method for multimodal data as described in any one of claims 1 to 7.
16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the label revision method for multimodal data as described in any one of claims 1 to 7.