Multi-modal noise tag correction method, equipment, device and medium
By constructing semantically invariant prototype representations of multimodal samples and a soft label correction strategy based on historical predictions, the problem of noisy labels in multimodal learning is solved, label quality is improved, and the reliability and generalization of multimodal data training are ensured.
Patent Information
- Application Number
- CN202610097852.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-24
- Publication Date
- 2026-04-21
AI Technical Summary
Noise labeling is a common problem in multimodal learning, which leads to deep models overfitting noisy data patterns, making it difficult to correctly learn the distribution of real data, and the error propagation phenomenon seriously affects the performance of downstream tasks.
By constructing semantically invariant prototype representations of multimodal samples, the association between multimodal samples and prototypes is used to filter noisy labels, and a soft label correction strategy combining historical predictions is adopted to correct noisy labels, thereby improving label quality.
Effectively filter out noisy labels to avoid misleading downstream multimodal learning models with incorrectly labeled data, thus ensuring the reliability and generalization of the multimodal data training process.
Smart Images
Figure CN121901853A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data noise processing technology, and in particular to a method, device, apparatus and medium for multimodal noise label correction based on prototype learning. Background Technology
[0002] With the rapid development of sensing and artificial intelligence technologies, data in critical fields such as intelligent manufacturing and healthcare is shifting from traditional unimodal to multimodal. Generally, the same thing or object can be described from different angles or through different approaches; these different descriptions constitute multimodality. Compared to traditional unimodal learning, multimodal learning incorporates more information from the original target, compensating for cognitive biases caused by insufficient information in unimodal learning. This effectively mitigates the impact of adverse factors and achieves better performance than unimodal learning.
[0003] The success of deep neural networks relies on high-quality labeled training data, but creating large-scale, well-labeled cross-modal datasets is extremely difficult, if not impossible. To reduce over-reliance on domain-specific data, the current mainstream learning paradigm is "pre-training-fine-tuning," which typically involves pre-training the model using large amounts of multimodal data collected from the internet, and then fine-tuning the model using domain-specific multimodal data. However, datasets obtained or created from real-world environments face the challenge of noise from various modal relationships, directly impacting the quality of multimodal data.
[0004] For example, multimodal data collected in complex application environments is easily affected by external environmental factors, such as poor weather or complex backgrounds, which can mislead annotators into giving incorrect labels to the modal data, i.e., noisy labels. The problem of noisy labels is prevalent in multimodal learning, mainly due to factors such as high data annotation costs, varying levels of annotator skill, and complex data sources. For instance, in large-scale image and text datasets, annotation errors, inconsistencies, or incompleteness occur frequently, and noisy label errors are generally difficult to detect.
[0005] Currently, most research aims to reduce the impact of noisy labels on model training by adjusting the loss function. However, these methods typically require complex loss function designs and may exhibit inconsistent performance across different datasets. Other works focus on sample selection, identifying and filtering potentially clean samples for training to avoid the model being misled by noisy samples. For example, methods like MentorNet and Co-teaching attempt to automatically select true labeled samples for learning. However, these methods are difficult to apply directly in multimodal scenarios because the data characteristics of different modalities vary significantly, increasing the difficulty of sample selection and preventing the effective use of mislabeled data. Summary of the Invention
[0006] In view of the above problems, the present invention provides a multimodal noise label correction method, device, apparatus, and medium to overcome or at least partially solve the above problems. It addresses the issue that multimodal data labels often contain some noise labels due to the influence of labeling algorithm accuracy and human labeling errors. Noisy labels easily cause deep models to overfit noisy data patterns, making it difficult to correctly learn the true data distribution. Furthermore, erroneous interactions between multimodal data can trigger complex error propagation phenomena, leading to performance degradation in downstream multimodal learning tasks.
[0007] This invention provides the following solution: A multimodal noise label correction method includes: The process involves acquiring multimodal data to be processed, using the encoder corresponding to each modal data to process each modal data, and obtaining a unified representation of each modality in the feature space. Obtain the prototype representation of each semantic category in each modality data to obtain the multimodal sample prototype; The distance between the sample in each modality data and the prototype of the multimodal sample is calculated, and the distance is combined with a threshold method to filter and obtain multimodal noise labels; A soft label correction strategy that combines historical predictions is used to correct the multimodal noise labels so that the noise labels are filtered and corrected by the prototype representation with semantic invariance.
[0008] Preferably, during training, a momentum update mechanism is used to update the multimodal sample prototype, and the momentum update mechanism is expressed by the following formula:
[0009] In the formula: This represents the new prototype representation generated during the training process. Indicates the first The first mode The prototype representation of each category, This represents the parameter that controls the intensity of momentum update, and ⊙ represents the XOR operation.
[0010] Preferably, the distance between a sample in each modal data and the prototype of the multimodal sample is measured by cosine similarity. The Data With the Matching degree of each class It can be expressed by the following formula:
[0011] In the formula: This represents the natural exponential function. Representing modes The Data With the Cosine similarity of each class , This represents the temperature coefficient.
[0012] Preferably, the soft-label correction strategy that combines historical predictions is expressed by the following formula:
[0013] In the formula: This indicates the corrected noise label. Modal The Data With the Noise labels for each category, This represents a parameter used to control the degree to which historical forecasts are retained. Representing modes The The degree of matching of the data.
[0014] Preferably, the loss function used in network training is expressed by the following formula:
[0015] In the formula: This represents the loss function used to narrow the distance between similar prototypes. This represents the robustness control loss function.
[0016] Preferably: a loss function used to narrow the distance between similar prototypes. It can be expressed by the following formula:
[0017] In the formula: Represents the number of modes. Indicates the first Number of samples in each modality Representing modes The One data point, Indicates the first Labeling categories for each modality The sample prototype representation, Indicates the first The first mode Prototype representation of each category.
[0018] Preferably: Robust control loss function It can be expressed by the following formula,
[0019] In the formula: Represents the number of modes. Indicates the first Number of samples in each modality Representing modes The Data With the Noise labels for each category, Representing modes The Data With the The degree of matching between classes.
[0020] A multimodal noise label correction device is provided for performing the above-described multimodal noise label correction method, the device comprising: The data acquisition unit is used to acquire the multimodal data to be processed, and to process each modal data using the encoder corresponding to each modal data to obtain a unified representation of each modality in the feature space. The sample prototype acquisition unit is used to obtain the prototype representation of each semantic category in each modality data to obtain the multimodal sample prototype; The noise label filtering unit is used to calculate the distance between the sample in each modality data and the prototype of the multimodal sample, and use the distance combined with a threshold method to filter and obtain multimodal noise labels; The noise label correction unit is used to correct the multimodal noise labels using a soft label correction strategy that incorporates historical predictions, so that the noise labels are filtered and corrected by the prototype representation with semantic invariance.
[0021] A multimodal noise tag correction device, the device comprising a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the above-described multimodal noise label correction method according to the instructions in the program code.
[0022] A computer-readable storage medium for storing program code for performing the above-described multimodal noise label correction method.
[0023] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This application provides a method, device, apparatus, and medium for multimodal noise label correction. The method filters noise labels by constructing semantically invariant prototype representations of multimodal samples and corrects noise labels by utilizing the correlation between multimodal samples and prototypes, thereby improving label quality and preventing erroneous label data from misleading downstream multimodal learning models, thus ensuring the reliability and generalization of multimodal data training. Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0025] Figure 1 This is a flowchart of a multimodal noise tag correction method provided in an embodiment of the present invention; Figure 2 This is a roadmap for the multimodal noise label correction technology for prototype discrimination learning provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of a multimodal noise tag correction device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a multimodal noise tag correction device provided in an embodiment of the present invention. Detailed Implementation
[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.
[0027] See Figure 1 This invention provides a multimodal noise label correction method, such as... Figure 1 As shown, the method may include: S101: Acquire the multimodal data to be processed, use the encoder corresponding to each modal data to process each modal data, and obtain a unified representation of each modality in the feature space; S102: Obtain the prototype representation of each semantic category in each modality data to obtain the multimodal sample prototype; as the model trains, the constructed prototype also needs to be updated, therefore a momentum update mechanism is introduced to ensure the stability of the training process. Specifically, the momentum update mechanism is used to update the multimodal sample prototype during training, and the momentum update mechanism is expressed by the following formula:
[0028] In the formula: This represents the new prototype representation generated during the training process. Indicates the first The first mode The prototype representation of each category, This represents the parameter that controls the intensity of momentum update, and ⊙ represents the XOR operation.
[0029] S103: Calculate the distance between samples in each modality data and the multimodal sample prototype, and use the distance combined with a threshold method for filtering to obtain multimodal noise labels; in specific implementation, embodiments of this application can provide that the distance between samples in each modality data and the multimodal sample prototype is measured by cosine similarity, modality The Data With the Matching degree of each class It can be expressed by the following formula:
[0030] In the formula: This represents the natural exponential function. Representing modes The Data With the Cosine similarity of each class , This represents the temperature coefficient.
[0031] S104: A soft label correction strategy combining historical predictions is used to correct the multimodal noise labels, so that the noise labels are filtered and corrected by the semantically invariant prototype representation. Specifically, embodiments of this application can provide a soft label correction strategy combining historical predictions, expressed by the following formula:
[0032] In the formula: This indicates the corrected noise label. Modal The Data With the Noise labels for each category, This represents a parameter used to control the degree to which historical forecasts are retained. Representing modes The The degree of matching of the data.
[0033] To achieve better training results, embodiments of this application may provide a loss function used in network training expressed by the following formula:
[0034] In the formula: This represents the loss function used to narrow the distance between similar prototypes. This represents the robustness control loss function.
[0035] Loss function used to narrow the distance between similar prototypes It can be expressed by the following formula:
[0036] In the formula: Represents the number of modes. Indicates the first Number of samples in each modality Representing modes The One data point, Indicates the first Labeling categories for each modality The sample prototype representation, Indicates the first The first mode Prototype representation of each category.
[0037] Robust control loss function It can be expressed by the following formula,
[0038] In the formula: Represents the number of modes. Indicates the first Number of samples in each modality Representing modes The Data With the Noise labels for each category, Representing modes The Data With the The degree of matching between classes.
[0039] The multimodal noise label correction method provided in this application is based on the prototype learning concept. It mines semantically invariant features of each modality through prototype learning, filters noisy labeled samples using prototypes, and corrects the noise labels of samples by leveraging the correlation between multimodal data, thereby improving the label quality of multimodal data. This method mainly includes three improvements: multimodal sample prototype construction, prototype-based multimodal noise label correction mechanism, and multimodal noise label prototype loss.
[0040] The multimodal noise tag correction method provided in the embodiments of this application will be described in detail below.
[0041] like Figure 2 As shown, this application embodiment uses an open-source pre-trained model as the encoder for each modality to obtain a unified representation of each modality in the feature space. By introducing a clustering algorithm, the prototype representation of each semantic category in each modality is obtained to preserve the invariance of modality and semantics, thereby effectively identifying noisy labels. For modalities... The mean of samples within a certain neighborhood of each class is used as its prototype sample.
[0042] For K modalities and C categories, all involved prototypes are represented as follows:
[0043] As the model trains, the prototype also needs to be updated. Therefore, a momentum update mechanism is introduced to ensure the stability of the training process. The momentum update process can be defined as follows:
[0044] In the formula: This represents the new prototype representation generated during the training process. Indicates the first The first mode The prototype representation of each category, This represents the parameter that controls the intensity of momentum update, and ⊙ represents the XOR operation.
[0045] Once the prototype is built, the distance between samples in each modality and the prototype is used to identify noise labels. Specifically, for each modality... The Data The distance between it and each prototype in that modality is measured by cosine similarity, i.e. Its relationship with the first The degree of matching for each class is defined as follows:
[0046] In the formula: This represents the natural exponential function. Representing modes The Data With the Cosine similarity of each class , This represents the temperature coefficient, used to control the stiffness of the training process.
[0047] To identify noisy labels, this application employs a threshold method for filtering, that is, comparing the sample with the labeled category. The prototype matching degree is less than These are then considered noise labels. To correct these noise labels, this application employs a soft label correction strategy that incorporates historical predictions, the process of which can be described as follows:
[0048] In the formula: This indicates the corrected noise label. Modal The Data With the Noise labels for each category, This represents a parameter used to control the degree to which historical forecasts are retained. Representing modes The The degree of matching of the data.
[0049] As training progresses, noisy labels are gradually filtered and corrected by semantically invariant prototype representations, thus enabling robust cross-modal learning.
[0050] Multimodal noise label prototype loss, by utilizing semantically invariant prototypes to correct multimodal noise labels, can mitigate the impact of erroneous labels within each modality, but still lacks interaction and collaboration between multiple modalities. Prototype representations of each modality should possess similar semantic characteristics to alleviate modal heterogeneity. Furthermore, prototype training should be robust during multi / cross-modal training. Specifically, the aim is to narrow the distance between similar prototypes to mitigate intermodal heterogeneity, and its loss function is as follows:
[0051] In the formula: Represents the number of modes. Indicates the first Number of samples in each modality Representing modes The One data point, Indicates the first Labeling categories for each modality The sample prototype representation, Indicates the first The first mode Prototype representation of each category.
[0052] To ensure robustness in prototype training, this application employs a robust loss model to satisfy noise risk theory, whereby the noise risk Rξ(L(f(x)) and the clean risk R(L(f(x)) share a global minimum f′. Its basic form is:
[0053] In the formula: Represents the number of modes. Indicates the first Number of samples in each modality Representing modes The Data With the Noise labels for each category, Representing modes The Data With the The degree of matching between classes.
[0054] Finally, the multimodal noise label learning loss function used for prototype learning is:
[0055] In the formula: This represents the loss function used to narrow the distance between similar prototypes. This represents the robustness control loss function.
[0056] In summary, the multimodal noise label correction method provided in this application filters noise labels by constructing semantically invariant prototype representations of multimodal samples and corrects noise labels by utilizing the correlation between multimodal samples and prototypes, thereby improving label quality and preventing erroneous label data from misleading downstream multimodal learning models, thus ensuring the reliability and generalization of multimodal data training.
[0057] See Figure 3 The present invention can also provide a multimodal noise tag correction device, such as... Figure 3 As shown, the apparatus for performing the above-described multimodal noise label correction method may include: The data acquisition unit 301 is used to acquire the multimodal data to be processed, and to process each modal data using the encoder corresponding to each modal data to obtain a unified representation of each modality in the feature space. The sample prototype acquisition unit 302 is used to acquire the prototype representation of each semantic category in each modality data to obtain the multimodal sample prototype; The noise label filtering unit 303 is used to calculate the distance between the sample in each modality data and the prototype of the multimodal sample, and use the distance combined with a threshold method to filter and obtain multimodal noise labels; The noise label correction unit 304 is used to correct the multimodal noise labels using a soft label correction strategy that incorporates historical predictions, so that the noise labels are filtered and corrected by the prototype representation with semantic invariance.
[0058] This invention can also provide a multimodal noise tag correction device, the device including a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of the multimodal noise label correction method described above according to the instructions in the program code.
[0059] like Figure 4 As shown in the figure, a multimodal noise tag correction device provided in this embodiment of the invention may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 all communicate with each other through the communication bus 13.
[0060] In this embodiment of the invention, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.
[0061] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiments of the multimodal noise label correction method.
[0062] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 11 stores at least a program for implementing the following functions: The process involves acquiring multimodal data to be processed, using the encoder corresponding to each modal data to process each modal data, and obtaining a unified representation of each modality in the feature space. Obtain the prototype representation of each semantic category in each modality data to obtain the multimodal sample prototype; The distance between the sample in each modality data and the prototype of the multimodal sample is calculated, and the distance is combined with a threshold method to filter and obtain multimodal noise labels; A soft label correction strategy that combines historical predictions is used to correct the multimodal noise labels so that the noise labels are filtered and corrected by the prototype representation with semantic invariance.
[0063] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.
[0064] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.
[0065] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.
[0066] Of course, it should be noted that, Figure 4 The structure shown does not constitute a limitation on the multimodal noise tag correction device in the embodiments of the present invention. In practical applications, the multimodal noise tag correction device may include more than Figure 4 More or fewer components as shown, or combinations of certain components.
[0067] This invention can also provide a computer-readable storage medium for storing program code for executing the steps of the multimodal noise label correction method described above.
[0068] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0069] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0070] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0071] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.
Claims
1. A method for correcting multimodal noise tags, characterized in that, The method includes: The process involves acquiring multimodal data to be processed, using the encoder corresponding to each modal data to process each modal data, and obtaining a unified representation of each modality in the feature space. Obtain the prototype representation of each semantic category in each modality data to obtain the multimodal sample prototype; The distance between the sample in each modality data and the prototype of the multimodal sample is calculated, and the distance is combined with a threshold method to filter and obtain multimodal noise labels; A soft label correction strategy that combines historical predictions is used to correct the multimodal noise labels so that the noise labels are filtered and corrected by the prototype representation with semantic invariance.
2. The multimodal noise tag correction method according to claim 1, characterized in that, During training, a momentum update mechanism is used to update the prototype of the multimodal samples. This momentum update mechanism is expressed by the following formula: In the formula: This represents the new prototype representation generated during the training process. Indicates the first The first mode The prototype representation of each category, This represents the parameter that controls the intensity of momentum update, and ⊙ represents the XOR operation.
3. The multimodal noise tag correction method according to claim 1, characterized in that, The distance between a sample in each modal data and the prototype of the multimodal sample is measured by cosine similarity. The Data With the Matching degree of each class It can be expressed by the following formula: In the formula: This represents the natural exponential function. Representing modes The Data With the Cosine similarity of each class , This represents the temperature coefficient.
4. The multimodal noise tag correction method according to claim 1, characterized in that, The soft-label correction strategy based on historical predictions is expressed by the following formula: In the formula: This indicates the corrected noise label. Modal The Data With the Noise labels for each category, This represents a parameter used to control the degree to which historical forecasts are retained. Representing modes The The degree of matching of the data.
5. The multimodal noise tag correction method according to claim 1, characterized in that, The loss function used in network training is expressed by the following formula: In the formula: This represents the loss function used to narrow the distance between similar prototypes. This represents the robustness control loss function.
6. The multimodal noise tag correction method according to claim 5, characterized in that, Loss function used to narrow the distance between similar prototypes It can be expressed by the following formula: In the formula: Represents the number of modes. Indicates the first Number of samples in each modality Representing modes The One data point, Indicates the first Labeling categories for each modality The sample prototype representation, Indicates the first The first mode Prototype representation of each category.
7. The multimodal noise tag correction method according to claim 5, characterized in that, Robust control loss function It can be expressed by the following formula, In the formula: Represents the number of modes. Indicates the first Number of samples in each modality Representing modes The Data With the Noise labels for each category, Representing modes The Data With the The degree of matching between classes.
8. A multimodal noise tag correction device, characterized in that, The apparatus for performing the multimodal noise tag correction method according to any one of claims 1-7, the apparatus comprising: The data acquisition unit is used to acquire the multimodal data to be processed, and to process each modal data using the encoder corresponding to each modal data to obtain a unified representation of each modality in the feature space. The sample prototype acquisition unit is used to obtain the prototype representation of each semantic category in each modality data to obtain the multimodal sample prototype; The noise label filtering unit is used to calculate the distance between the sample in each modality data and the prototype of the multimodal sample, and to use the distance combined with a threshold method to filter and obtain multimodal noise labels; The noise label correction unit is used to correct the multimodal noise labels using a soft label correction strategy that incorporates historical predictions, so that the noise labels are filtered and corrected by the prototype representation with semantic invariance.
9. A multimodal noise tag correction device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the multimodal noise label correction method according to any one of claims 1-7 according to the instructions in the program code.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for performing the multimodal noise label correction method according to any one of claims 1-7.