An AI data precise labeling method and system based on multi-modal feature fusion

Abstract

This invention discloses a method and system for accurate AI data annotation based on multimodal feature fusion, relating to the field of artificial intelligence technology. The method includes: acquiring raw input data and encoding features for each modality to obtain initial features; calculating global context features based on the initial features, and inputting the global context features and initial features together into a dynamic gating network to generate modality fusion weights; weighting the initial features according to the modality fusion weights to obtain weighted modality features; extracting local fine-grained representations and global semantic representations from the weighted modality features, and performing cross-modality fine-grained alignment based on the local fine-grained representations to bring the semantically matching local fine-grained representations closer, constraining the overall semantic consistency between different modalities through the global semantic representation, and generating multi-granularity fusion features; performing category prediction based on the multi-granularity fusion features, outputting the category prediction results, and calculating the prediction uncertainty.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for accurate labeling of AI data through multimodal feature fusion. Background Technology

[0002] With the rapid development of deep learning and multimodal perception technologies, joint modeling based on multi-source heterogeneous data such as images, text, and speech has become a research hotspot in the field of artificial intelligence. Multimodal pre-trained models, through contrastive learning or generative alignment strategies, have demonstrated powerful capabilities in cross-modal semantic understanding tasks. Modal fusion methods have gradually evolved from early feature splicing and weighted averaging to dynamic interaction and fine-grained alignment based on attention mechanisms, improving the performance of models in scenarios such as image and text retrieval, visual question answering, and cross-modal generation.

[0003] However, modality fusion often uses static weights or black-box cross-attention, which makes it difficult to dynamically adjust the contribution of each modality based on sample-level semantic consistency. This can easily lead to fusion bias, especially when there is noise or missing data in a certain modality. Uncertainty estimation usually only relies on the probability distribution of the final prediction or the model perturbation variance, failing to fully decouple different error sources such as sample ambiguity, model cognitive limitations and semantic conflicts between modalities. Furthermore, it cannot identify the high-risk situation where multiple modalities consistently output erroneous results. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for accurate AI data annotation based on multimodal feature fusion to address the difficulty in dynamically adjusting the contribution of each modality based on sample-level semantic consistency. This is especially problematic when noise or missing data in a particular modality can lead to fusion bias. Uncertainty estimation typically relies solely on the probability distribution of the final prediction or the model perturbation variance, failing to adequately decouple different error sources such as sample ambiguity, model cognitive limitations, and semantic conflicts between modalities. Furthermore, it cannot identify the high-risk situation where multiple modalities consistently output incorrect results.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for accurate labeling of AI data through multimodal feature fusion, which includes acquiring raw input data and encoding features for each modality to obtain initial features; Global context features are calculated based on initial features, and the global context features and initial features are input together into a dynamic gating network to generate modality fusion weights. The initial features are weighted according to the modality fusion weights to obtain weighted modality features; Local fine-grained representations and global semantic representations are extracted from weighted modal features, and cross-modal fine-grained alignment is performed based on the local fine-grained representations to bring the local fine-grained representations of semantic matching closer. The overall semantic consistency between different modalities is constrained by the global semantic representation to generate multi-granularity fusion features. Class prediction is performed based on multi-granularity fusion features, the class prediction results are output and the prediction uncertainty is calculated; Based on the uncertainty of the prediction, the original labels are adjusted with confidence weight to generate optimized labels; Starting with the original input data, feature encoding, dynamic gating network processing, weighted fusion, cross-modal semantic alignment and classification prediction are performed to form a labeled model, and the labeled model is iteratively trained using optimized labeled labels.

[0007] As a preferred embodiment of the AI ​​data accurate annotation method for multimodal feature fusion described in this invention, the specific steps for obtaining the original input data and encoding the features of each modality to obtain initial features are as follows: Acquire raw input data containing at least two types of data, including images, text, or speech; The image modality, text modality, and speech modality are input into the visual encoder, speech encoder, and audio encoder, respectively, to obtain the initial feature sequence of the image, the initial feature sequence of the text, and the initial feature sequence of the speech. The initial feature sequences of images, text, and speech are collectively referred to as initial features.

[0008] As a preferred embodiment of the AI ​​data accurate annotation method for multimodal feature fusion described in this invention, the specific steps of calculating global context features based on initial features and inputting the global context features and initial features together into a dynamic gating network to generate modality fusion weights are as follows: Extract the global semantic vector of each modality from the initial features corresponding to each modality; The global semantic vectors of all modalities are arithmetically averaged to obtain a global context feature vector with the same dimension as the global semantic vectors of each modality. The global context features are concatenated with the global semantic vector of each modality to form a joint feature vector; The joint feature vector is input into a dynamic gating network consisting of a single-layer linear transformation and a Sigmoid activation function to calculate the original gating response value. The original gated response values ​​are normalized to generate modal fusion weights.

[0009] As a preferred embodiment of the AI ​​data accurate annotation method for multimodal feature fusion described in this invention, the steps of performing cross-modal fine-grained alignment based on local fine-grained representation, bringing the local fine-grained representation of semantic matching closer, and constraining the overall semantic consistency between different modalities through global semantic representation to generate multi-granularity fusion features are as follows: For the initial feature sequence of each modality, the corresponding modality fusion weights are weighted element by element to obtain the weighted modality feature sequence. From the weighted modal feature sequence, the feature at the first position is extracted as a global semantic representation. The feature at the first position has been configured as a dedicated representation unit for aggregating the overall semantic information of the modality during the modal coding stage, and the features at the remaining positions constitute local region representations. For image and text modalities, the dot product between the feature vector of each image region and the feature vector of each text word is calculated as the original association strength. The original association strength is then normalized according to the image region dimension to generate cross-modal local matching weights. Based on the cross-modal local matching weights, the local features of the image are weighted and fused: based on the original association strength corresponding to the cross-modal local matching weights, the association strength between each image region and all text words is summed to obtain the semantic reliability score of the image region. The semantic reliability score is used as a weighting coefficient to scale the features of the image region, thereby adjusting its contribution in local fusion, reducing the influence of background or noise regions that are unrelated to the text description, enhancing semantically consistent local features, and obtaining local fused features. The global semantic representations of all modalities are arithmetically averaged to obtain the overall semantic center. The overall semantic center is then weighted and combined with the alignment-enhanced local fusion features. The contributions of global semantic consistency and local semantic alignment in the fusion features are balanced by the local enhancement coefficients, thereby generating multi-granularity fusion features.

[0010] As a preferred embodiment of the AI ​​data accurate annotation method based on multimodal feature fusion described in this invention, the specific steps of performing category prediction based on multi-granularity fusion features, outputting category prediction results, and calculating prediction uncertainty are as follows: The multi-granularity fusion features are input into the classification module, and the fusion prediction results are output after probability normalization. The global semantic representations of each modality are input into the same classification module to obtain independent prediction results for each modality; With the random deactivation mechanism enabled, forward inference is repeatedly performed on the multi-granularity fusion features to obtain a set of prediction results with random perturbations. Based on the fused prediction results, the independent prediction results of each modality, and the set of prediction results obtained from multiple forward inferences, the prediction uncertainty is calculated comprehensively: Information entropy is calculated based on the probability distribution of the fusion prediction results to reflect the degree of discrimination ambiguity of the current sample itself; The statistical variance is calculated based on the degree of fluctuation in the results of multiple forward inferences to reflect the uncertainty of the model's understanding of the sample. Inconsistency measures are calculated based on the distribution differences between the independent prediction results of each modality to reflect the degree of conflict between different modalities at the high-level semantic level. When the inconsistency metric is lower than a preset divergence threshold, an external knowledge consistency check is introduced: The category names in the independent prediction results of each modality are input into a pre-trained and parameter-frozen external text semantic encoder to obtain the predicted semantic representation. The raw input data of each modality is input to the external modality encoder respectively. The image input is to the external image encoder, the text input is to the external text encoder, and the speech is converted into text by the fixed speech recognition model and then input to the external text encoder to obtain the input semantic representation. Calculate the cosine similarity between each pair of predicted semantic representations and input semantic representations; If any cosine similarity is lower than the preset consistency threshold, the independent prediction of the modality is determined to lack semantic support. If all independent predictions of all modalities are deemed to lack semantic support, then the basic uncertainty will be forcibly set to a preset lower limit as an indicator of prediction uncertainty.

[0011] As a preferred embodiment of the AI ​​data accurate annotation method based on multimodal feature fusion described in this invention, the step of performing confidence-weighted correction on the original annotation labels according to prediction uncertainty to generate optimized annotation labels includes the following steps: Set an uncertainty threshold; The prediction uncertainty index is compared with the uncertainty threshold, and the label softening coefficient is calculated based on the comparison result. The label softening coefficient is a non-negative number not exceeding 1, and increases with the increase of the prediction uncertainty index. The original hard-labeled labels are combined with the fused predicted probability distribution to generate optimized labels.

[0012] As a preferred embodiment of the AI ​​data accurate annotation method based on multimodal feature fusion described in this invention, the steps include: starting with the original input data, performing feature encoding, dynamic gating network processing, weighted fusion, cross-modal semantic alignment, and classification prediction to construct an annotation model, and iteratively training the annotation model using optimized annotation labels. The specific steps are as follows: The multimodal input encoding module, the global context-guided dynamic gating module, the multi-granularity cross-modal alignment and fusion module, and the multi-source prediction and uncertainty quantification module are sequentially connected and parameterized into a neural network structure, wherein differentiable operations are used within each module and between modules, thereby forming an end-to-end labeled model. Based on the optimized label and the predicted probability distribution of the model output, calculate the cross-entropy loss; Update all learnable parameters of the labeled model using the backpropagation algorithm; Repeat the entire closed-loop process from acquiring raw input data to optimizing label generation and updating model parameters until the loss converges, thus completing the self-evolutionary iterative training.

[0013] Secondly, the present invention provides an AI data accurate labeling system with multimodal feature fusion, including an input encoding module, a gating weight module, a feature weighting module, a multi-granularity fusion module, an uncertainty calculation module, a label optimization module, and an iterative training module; The input encoding module acquires the original input data and performs feature encoding on each modality to obtain initial features; The gating weight module calculates global context features based on initial features, and inputs the global context features and initial features into the dynamic gating network to generate modality fusion weights. The feature weighting module weights the initial features according to the modality fusion weights to obtain weighted modal features; The multi-granularity fusion module extracts local fine-grained representations and global semantic representations from the weighted modal features, performs cross-modal fine-grained alignment based on the local fine-grained representations, brings the local fine-grained representations of semantic matching closer, and constrains the overall semantic consistency between different modalities through the global semantic representation to generate multi-granularity fusion features. The uncertainty calculation module performs category prediction based on multi-granularity fusion features, outputs the category prediction result, and calculates the prediction uncertainty. The label optimization module performs confidence-weighted correction on the original labels based on the prediction uncertainty to generate optimized labels. The iterative training module starts with the original input data and performs feature encoding, dynamic gating network processing, weighted fusion, cross-modal semantic alignment and classification prediction to form a labeled model. The module then uses optimized labels to iteratively train the labeled model.

[0014] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the AI ​​data accurate annotation method for multimodal feature fusion as described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the AI ​​data accurate annotation method for multimodal feature fusion as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: It explicitly separates local fine-grained representation and global semantic representation from weighted modal features, calculates the semantic reliability score of image regions based on the original dot product correlation strength, selectively enhances and suppresses noise in local features, and uses the mean of all modal global representations as the overall semantic center to constrain cross-modal consistency, thereby generating multi-granular fusion features that contain both fine-grained matching evidence and global semantic consensus. Based on the fusion features, it simultaneously performs fusion prediction, modal independent prediction, and multiple perturbation inference, integrates three components: information entropy, statistical variance, and inter-modal distribution differences, and introduces an external pre-trained encoder for semantic similarity verification in low-disagreement scenarios to achieve proactive identification of hidden errors such as consistent ground faults, and outputs an operable uncertainty index that integrates internal state and external common sense rationality. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart for an AI data accurate annotation method based on multimodal feature fusion; Figure 2 A schematic diagram of an AI data accurate annotation system that integrates multimodal feature fusion. Detailed Implementation

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0022] Reference Figures 1-2 This is one embodiment of the present invention, which provides a method for accurate AI data annotation based on multimodal feature fusion, comprising the following steps: S1. Obtain the original input data and encode the features of each modality to obtain the initial features.

[0023] Furthermore, raw input data containing at least two types of data, including images, text, or speech, is collected.

[0024] The image modality, text modality, and speech modality are input into the visual encoder, speech encoder, and audio encoder, respectively, to obtain the initial feature sequence of the image, the initial feature sequence of the text, and the initial feature sequence of the speech.

[0025] The initial feature sequences of images, text, and speech are collectively referred to as initial features.

[0026] It should be noted that by using dedicated encoders to extract features from different modalities such as images, text, and speech, and unifying the output into a sequence form of the same dimension, the semantic information of each modality retains its original structure while possessing the mathematical consistency required for cross-modal interaction. This provides a clear and dimensionally aligned initial feature input for subsequent dynamic fusion and fine-grained alignment.

[0027] The reason for using a modal-specific encoder instead of a unified encoder is that images, text, and speech have fundamental differences in signal morphology, temporal structure, and semantic granularity: images have spatial locality, which is suitable for convolution or ViT modeling; text depends on word order and context, which requires autoregression or bidirectional attention; and speech is a time-frequency signal, which needs to take into account both temporal dynamics and spectral features.

[0028] Forcing the use of the same network to process all modalities will lead to distortion of semantic representation.

[0029] By extracting features from the optimal architecture in their respective domains and mapping them to a unified latent space, modal characteristics are respected and a comparable basis is provided for subsequent alignment.

[0030] It is not a simple patching of existing models, but a collaborative coding paradigm built on the principles of intramodal fidelity and intermodal comparability, which is a prerequisite for achieving high-precision multimodal annotation.

[0031] S2. Calculate global context features based on initial features, and input the global context features and initial features together into the dynamic gating network to generate modality fusion weights.

[0032] Furthermore, the global semantic vector of each modality is extracted from the initial features corresponding to each modality.

[0033] The global semantic vectors of all modalities are arithmetically averaged to obtain a global context feature vector with the same dimension as the global semantic vectors of each modality.

[0034] The global contextual features are concatenated with the global semantic vector of each modality to form a joint feature vector.

[0035] The joint feature vector is input into a dynamic gating network consisting of a single-layer linear transformation and a Sigmoid activation function to calculate the original gating response value.

[0036] The original gated response values ​​are normalized to generate modal fusion weights.

[0037] It should be noted that global contextual features are obtained by extracting global semantic vectors from the initial features of each modality and calculating the arithmetic mean. Then, the contextual features are concatenated with the global representation of each modality and input into the dynamic gating network, so that the fusion weights of each modality can show the degree of consistency between perception and overall semantics.

[0038] The generated modality fusion weights are normalized scalar values, which have clear physical meaning and can be directly used for subsequent feature weighting, ensuring the interpretability and computational efficiency of the gating mechanism.

[0039] In some samples, the speech may be clear but the image is blurry; in other samples, the text description may conflict with the image content.

[0040] Therefore, the fusion weights must be adaptive to the samples. If they rely solely on the judgment of a single modality, they are prone to getting trapped in local optima. By introducing a global context as a reference anchor, the gating network can answer whether the modality is consistent with the majority of modalities, thereby suppressing abnormal modalities.

[0041] The arithmetic mean was chosen over complex aggregation to avoid introducing additional uncontrollable parameters and to ensure that the gating is lightweight, stable, and differentiable.

[0042] This individual-group comparison mechanism achieves dynamic credibility assessment at the semantic level while maintaining low computational overhead.

[0043] S3. Weight the initial features according to the modality fusion weights to obtain weighted modality features.

[0044] Furthermore, the weighting operation uses modal-level scalar weights to scale the initial feature sequences of each modality position by position. That is, each modality corresponds to a fusion weight scalar, and the scalar is multiplied by each local feature vector in the initial feature sequence of the modality. In this way, the overall semantic contribution of the modality is adaptively adjusted while preserving the original feature structure and dimension.

[0045] It should be noted that by using modal-level scalar weights to scale the entire initial feature sequence position by position, the overall confidence of the modality can be continuously adjusted without changing the internal structure of the features.

[0046] The weighting method has fewer parameters, stable gradient propagation, and is compatible with the assumption that the first position is used as a global representation. This ensures the consistency of the semantic role of the feature flow before and after weighting, providing a reliable basic representation for multi-granularity fusion.

[0047] Whether a modality is reliable usually depends on its overall quality rather than local regions. Therefore, there is no need to assign independent weights to each feature location; a single scalar can be used to express the overall usability of the modality.

[0048] The mainstream Transformer architecture has agreed that the first position carries global semantics. If the sequence is not weighted uniformly, the agreement will be broken.

[0049] The use of uniform scalar scaling preserves semantic integrity and avoids the instability of local weight estimation. This minimalist design achieves a balance between expressiveness and generalization, laying a structural foundation for subsequent local-global separation.

[0050] S4. Extract local fine-grained representations and global semantic representations from the weighted modal features respectively, and perform cross-modal fine-grained alignment based on the local fine-grained representations to bring the local fine-grained representations of semantic matching closer. Constrain the overall semantic consistency between different modalities through the global semantic representation to generate multi-granularity fusion features.

[0051] Furthermore, for the initial feature sequence of each modality, the corresponding modality fusion weights are weighted element by element to obtain the weighted modality feature sequence.

[0052] From the weighted modal feature sequence, the feature at the first position is extracted as the global semantic representation. The feature at the first position has been configured as a dedicated representation unit for aggregating the overall semantic information of the modality during the modality encoding stage, and the features at the remaining positions constitute the local region representation.

[0053] For both image and text modalities, the dot product between the feature vector of each image region and the feature vector of each text word is calculated as the original association strength. The original association strength is then normalized according to the image region dimension to generate cross-modal local matching weights.

[0054] Based on cross-modal local matching weights, weighted fusion of local image features is performed: based on the original association strength corresponding to the cross-modal local matching weights, the association strength between each image region and all text words is summed to obtain the semantic reliability score of the image region. The semantic reliability score is used as a weighting coefficient to scale the features of the image region, thereby adjusting its contribution in local fusion, reducing the influence of background or noise regions that are unrelated to the text description, enhancing semantically consistent local features, and obtaining local fused features.

[0055] The global semantic representations of all modalities are arithmetically averaged to obtain the overall semantic center. The overall semantic center is then weighted and combined with the alignment-enhanced local fusion features. The contributions of global semantic consistency and local semantic alignment in the fusion features are balanced by the local enhancement coefficients, thereby generating multi-granularity fusion features.

[0056] It should be noted that by fixing the first position of the weighted feature sequence as the global semantic representation and calculating the semantic reliability score of the image region based on the original dot product correlation strength, the local fusion process can effectively distinguish semantically related regions from background noise.

[0057] By weighting and combining the alignment-enhanced local features with the mean of the global representations of all modalities, the resulting multi-granularity fusion features contain both fine-grained matching evidence and cross-modal semantic consensus. These features can be directly used as input to the classification module without additional aggregation operations.

[0058] Traditional cross-attention methods weight all regions equally, leading to the integration of background information and diluting key semantics.

[0059] We deliberately trace back to the original, unnormalized dot product strength because the former preserves absolute relevance information. If a region has a small dot product with all text lexical units, it means that it is unrelated; otherwise, it is related.

[0060] By summing these original strengths to obtain a semantic reliability score, an objective scoring of local units is achieved, rather than a relative ranking.

[0061] Global semantics forms semantic centers through arithmetic averaging, imposing overall consistency constraints on local alignment to prevent overfitting of local matches.

[0062] This dual-track mechanism of local selective enhancement and global consistency anchoring enables the fused features to possess both discriminativeness and robustness.

[0063] S5. Perform category prediction based on multi-granularity fusion features, output the category prediction results and calculate the prediction uncertainty.

[0064] Furthermore, the multi-granularity fused features are input into the classification module, and the fused prediction results are output after probability normalization.

[0065] The global semantic representations of each modality are input into the same classification module to obtain independent prediction results for each modality.

[0066] With the random deactivation mechanism enabled, forward inference is repeatedly performed on multi-granularity fusion features to obtain a set of prediction results with random perturbations.

[0067] Based on the fused prediction results, the independent prediction results of each modality, and the set of prediction results obtained from multiple forward inferences, the prediction uncertainty is calculated comprehensively: The information entropy is calculated based on the probability distribution of the fusion prediction results to reflect the degree of ambiguity in the current sample itself.

[0068] The statistical variance is calculated based on the degree of fluctuation in the results of multiple forward inferences to reflect the uncertainty of the model's understanding of the sample.

[0069] Inconsistency measures are calculated based on the distribution differences between the independent prediction results of each modality to reflect the degree of conflict between different modalities at the high-level semantic level.

[0070] When the inconsistency metric falls below a preset divergence threshold, an external knowledge consistency check is introduced: The category names from the independent prediction results of each modality are input into a pre-trained external text semantic encoder with frozen parameters to obtain the predicted semantic representation.

[0071] The raw input data of each modality is input to the external modality encoder. The image input is to the external image encoder, the text input is to the external text encoder, and the speech is converted into text by the fixed speech recognition model and then input to the external text encoder to obtain the input semantic representation.

[0072] Calculate the cosine similarity between each pair of predicted semantic representations and input semantic representations.

[0073] If any cosine similarity is lower than a preset consistency threshold, the independent prediction of the determined modality lacks semantic support.

[0074] If all independent predictions for all modalities are deemed to lack semantic support, the basic uncertainty will be raised to a level no lower than a preset lower limit, which will serve as an indicator of prediction uncertainty.

[0075] By using two adjustable weighting coefficients, the three types of uncertainty components are weighted and fused together, and combined with the prediction uncertainty adjusted by external knowledge consistency verification, a unified prediction uncertainty index is generated.

[0076] The classification module is implemented using a single-layer fully connected neural network connected to the Softmax function. Multi-granularity fusion features Input to weight matrix Calculate the logits vector The fused prediction result is then obtained after Softmax normalization: ; For independent prediction of each modality, taking the image-text bimodal model as an example: Global semantic representation of image modalities Inputs are fed into the same classification module, and the output images are independently predicted to have different distributions. .

[0077] Global semantic representation of text modalities Similarly, the shared classification module is input, and the independent predicted distribution of the output text is obtained. . Ensure that predictions of each modality are comparable within the same semantic space.

[0078] During the uncertainty quantification phase, a random deactivation mechanism for Dropout is enabled, and Dropout remains active during the inference phase.

[0079] Repeat execution Each forward propagation generates slightly different logits due to the different Dropout masks, thus obtaining a set of perturbation prediction results. .

[0080] Based on this set, the prediction variance for each category can be calculated: ; in for The mean of the predictions is used to characterize the model's cognitive uncertainty. To measure the degree of volatility in category prediction results, The number of times the forward reasoning is repeated. No. Category in sub-fusion prediction The probability of.

[0081] It should be noted that by comprehensively integrating the information entropy of the prediction, the statistical variance of multiple inferences, and the inconsistency measure of the prediction distribution between modalities, and by introducing semantic similarity verification based on an external pre-trained encoder in low-disagreement scenarios, the prediction uncertainty not only reflects the internal state of the model, but also incorporates the rationality judgment of the external knowledge space.

[0082] When all modalities lack semantic support, the uncertainty is forcibly increased to a preset lower limit to ensure that the index can always effectively characterize the potential unreliability of the annotation results, and the output is a bounded real number that can be directly used for threshold judgment and coefficient calculation in subsequent label correction.

[0083] When multiple modes consistently output incorrect labels due to common bias, traditional uncertainty indicators will be very low, and the system will misjudge it as high confidence.

[0084] To address this, a triple uncertainty component is constructed to capture different sources of error.

[0085] When the modalities are highly consistent, external knowledge verification is initiated—using general models such as frozen CLIP as third-party judges to determine whether the predicted category matches the input content in the commonsense semantic space.

[0086] If there is a mismatch, it indicates that the system is trapped in a self-consistency illusion. In this case, forcibly increasing uncertainty is a safety fallback mechanism.

[0087] Extending uncertainty from internal measurement to joint internal and external verification improves the reliability boundary of the annotation system.

[0088] S6. Based on the prediction uncertainty, perform confidence-weighted correction on the original labels to generate optimized labels.

[0089] Furthermore, we can set an uncertainty threshold.

[0090] The prediction uncertainty index is compared with the uncertainty threshold, and the label softening coefficient is calculated based on the comparison results. The label softening coefficient is a non-negative number not exceeding 1, and it increases with the increase of the prediction uncertainty index.

[0091] The original hard-labeled labels are combined with the fused predicted probability distribution to generate optimized labels.

[0092] It should be noted that by comparing the prediction uncertainty with a preset threshold to calculate the label softening coefficient, and by performing a convex combination of the original hard label and the fused prediction distribution, the generated optimized label always satisfies the basic constraints of the probability distribution and can be directly used as the supervision target of the cross-entropy loss.

[0093] The labels retain the original supervision signals of high-confidence samples while introducing model self-feedback for low-confidence samples, forming a differentiable, trainable, and deployable soft supervision mechanism.

[0094] Traditional methods treat labels as fixed truth values ​​and cannot correct initial noise. They treat labels as soft targets that can be optimized, and their confidence level is dynamically adjusted by uncertainty.

[0095] When uncertainty is low, the original labels are almost completely trusted, and strong supervision is maintained. When uncertainty is high, the model turns to its own predictions to achieve self-correction.

[0096] Ensures that the output always follows a valid probability distribution, allowing for seamless integration with standard loss functions.

[0097] The softening coefficient increases monotonically with uncertainty, resulting in a smooth transition rather than a hard switch, avoiding training oscillations, and enabling the system to learn from noise.

[0098] S7. Starting with the original input data, feature encoding, dynamic gating network processing, weighted fusion, cross-modal semantic alignment and classification prediction are performed to form a labeling model, and the labeling model is iteratively trained using optimized label labels.

[0099] Furthermore, the multimodal input encoding module, the global context-guided dynamic gating module, the multi-granularity cross-modal alignment and fusion module, and the multi-source prediction and uncertainty quantification module are sequentially connected and parameterized into a neural network structure. Differentiable operations are used within and between each module to form an end-to-end labeled model.

[0100] The cross-entropy loss is calculated based on the optimized labeled and predicted probability distribution of the model output.

[0101] The backpropagation algorithm is used to update all learnable parameters of the labeled model.

[0102] Repeat the entire closed-loop process from acquiring raw input data to optimizing label generation and updating model parameters until the loss converges, thus completing the self-evolutionary iterative training.

[0103] It should be noted that multimodal coding, dynamic gating, multi-granularity fusion, uncertainty-aware prediction and other modules are integrated into a unified neural network architecture. All components are implemented using differentiable operations, which enables end-to-end backpropagation throughout the entire process from the original input to the optimized label.

[0104] By updating all learnable parameters with optimized labeling as a supervisory signal and repeatedly executing the closed-loop process, the system can continuously improve its labeling capabilities during training. The output model not only has classification capabilities but can also be directly deployed as an automated and accurate labeling tool in new data labeling tasks.

[0105] By incorporating the annotation process itself into a learnable framework, a positive cycle of "prediction → evaluation → correction → retraining" is formed.

[0106] Its feasibility depends on two points: first, all operations are continuously differentiable and support gradient backpropagation; second, the optimization label is a soft target, and its gradient can be propagated to the encoder and gating network.

[0107] The self-supervised and self-correcting coupled architecture enables the system to simultaneously improve annotation quality and model performance without external human intervention, converging to a highly robust automatic annotator.

[0108] This embodiment also provides an AI data accurate labeling system with multimodal feature fusion, including: an input encoding module, a gating weight module, a feature weighting module, a multi-granularity fusion module, an uncertainty calculation module, a label optimization module, and an iterative training module.

[0109] The input encoding module acquires the raw input data and performs feature encoding on each modality to obtain the initial features.

[0110] The gating weight module calculates global context features based on initial features, and inputs the global context features and initial features together into the dynamic gating network to generate modality fusion weights.

[0111] The feature weighting module weights the initial features according to the modality fusion weights to obtain weighted modal features.

[0112] The multi-granularity fusion module extracts local fine-grained representations and global semantic representations from the weighted modal features, performs cross-modal fine-grained alignment based on the local fine-grained representations, brings the local fine-grained representations of semantic matching closer, and constrains the overall semantic consistency between different modalities through the global semantic representation to generate multi-granularity fusion features.

[0113] The uncertainty calculation module performs category prediction based on multi-granularity fusion features, outputs the category prediction results, and calculates the prediction uncertainty.

[0114] The label optimization module performs confidence-weighted correction on the original labels based on the prediction uncertainty, generating optimized labels.

[0115] The iterative training module starts with the original input data and performs feature encoding, dynamic gating network processing, weighted fusion, cross-modal semantic alignment, and classification prediction to form a labeled model. The module then uses optimized labels to iteratively train the labeled model.

[0116] This embodiment also provides a computer device applicable to the AI ​​data accurate annotation method of multimodal feature fusion, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the AI ​​data accurate annotation method of multimodal feature fusion as proposed in the above embodiment.

[0117] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0118] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the AI ​​data accurate annotation method for multimodal feature fusion as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0119] In summary, this invention achieves the following: explicitly separating local fine-grained representations and global semantic representations from weighted modal features; calculating the semantic reliability score of image regions based on the original dot product correlation strength; selectively enhancing and suppressing noise in local features; and using the mean of all modal global representations as the overall semantic center to constrain cross-modal consistency. This generates multi-granular fusion features that contain both fine-grained matching evidence and global semantic consensus. Based on these fusion features, fusion prediction, modal independent prediction, and multiple perturbation inferences are performed simultaneously. The invention integrates three components: information entropy, statistical variance, and inter-modal distribution differences. In low-disagreement scenarios, an external pre-trained encoder is introduced to perform semantic similarity verification, enabling proactive identification of hidden errors such as consistent ground faults. Finally, an operable uncertainty index that integrates internal state and external common sense rationality is output.

[0120] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for accurate AI data annotation based on multimodal feature fusion, characterized in that: include: Obtain the raw input data and encode the features of each modality to obtain the initial features; Global context features are calculated based on initial features, and the global context features and initial features are input together into a dynamic gating network to generate modality fusion weights. The initial features are weighted according to the modality fusion weights to obtain weighted modality features; Local fine-grained representations and global semantic representations are extracted from weighted modal features, and cross-modal fine-grained alignment is performed based on the local fine-grained representations to bring the local fine-grained representations of semantic matching closer. The overall semantic consistency between different modalities is constrained by the global semantic representation to generate multi-granularity fusion features. Class prediction is performed based on multi-granularity fusion features, the class prediction results are output and the prediction uncertainty is calculated; Based on the uncertainty of the prediction, the original labels are adjusted with confidence weight to generate optimized labels; Starting with the original input data, feature encoding, dynamic gating network processing, weighted fusion, cross-modal semantic alignment and classification prediction are performed to form a labeled model, and the labeled model is iteratively trained using optimized labeled labels.

2. The AI ​​data accurate annotation method based on multimodal feature fusion as described in claim 1, characterized in that: The specific steps for obtaining the original input data and encoding the features of each modality to obtain the initial features are as follows: Acquire raw input data containing at least two types of data, including images, text, or speech; The image modality, text modality, and speech modality are input into the visual encoder, speech encoder, and audio encoder, respectively, to obtain the initial feature sequence of the image, the initial feature sequence of the text, and the initial feature sequence of the speech. The initial feature sequences of images, text, and speech are collectively referred to as initial features.

3. The AI ​​data accurate annotation method based on multimodal feature fusion as described in claim 2, characterized in that: The specific steps for calculating global context features based on initial features and inputting both global context features and initial features into a dynamic gating network to generate modality fusion weights are as follows: Extract the global semantic vector of each modality from the initial features corresponding to each modality; The global semantic vectors of all modalities are arithmetically averaged to obtain a global context feature vector with the same dimension as the global semantic vectors of each modality. The global context features are concatenated with the global semantic vector of each modality to form a joint feature vector; The joint feature vector is input into a dynamic gating network consisting of a single-layer linear transformation and a Sigmoid activation function to calculate the original gating response value. The original gated response values ​​are normalized to generate modal fusion weights.

4. The AI ​​data accurate annotation method based on multimodal feature fusion as described in claim 3, characterized in that: The method of cross-modal fine-grained alignment based on local fine-grained representation, bringing the local fine-grained representation of semantic matching closer together, and constraining the overall semantic consistency between different modalities through global semantic representation to generate multi-granularity fusion features, specifically includes the following steps: For the initial feature sequence of each modality, the corresponding modality fusion weights are weighted element by element to obtain the weighted modality feature sequence. From the weighted modal feature sequence, the feature at the first position is extracted as a global semantic representation. The feature at the first position has been configured as a dedicated representation unit for aggregating the overall semantic information of the modality during the modal coding stage, and the features at the remaining positions constitute local region representations. For image and text modalities, the dot product between the feature vector of each image region and the feature vector of each text word is calculated as the original association strength. The original association strength is then normalized according to the image region dimension to generate cross-modal local matching weights. Based on the cross-modal local matching weights, the local features of the image are weighted and fused: based on the original association strength corresponding to the cross-modal local matching weights, the association strength between each image region and all text words is summed to obtain the semantic reliability score of the image region. The semantic reliability score is used as a weighting coefficient to scale the features of the image region, thereby adjusting its contribution in local fusion, reducing the influence of background or noise regions that are unrelated to the text description, enhancing semantically consistent local features, and obtaining local fused features. The global semantic representations of all modalities are arithmetically averaged to obtain the overall semantic center. The overall semantic center is then weighted and combined with the alignment-enhanced local fusion features. The contributions of global semantic consistency and local semantic alignment in the fusion features are balanced by the local enhancement coefficients, thereby generating multi-granularity fusion features.

5. The AI ​​data accurate annotation method based on multimodal feature fusion as described in claim 4, characterized in that: The specific steps for class prediction based on multi-granularity fusion features, outputting class prediction results and calculating prediction uncertainty are as follows: The multi-granularity fusion features are input into the classification module, and the fusion prediction results are output after probability normalization. The global semantic representations of each modality are input into the same classification module to obtain independent prediction results for each modality; With the random deactivation mechanism enabled, forward inference is repeatedly performed on the multi-granularity fusion features to obtain a set of prediction results with random perturbations. Based on the fused prediction results, the independent prediction results of each modality, and the set of prediction results obtained from multiple forward inferences, the prediction uncertainty is calculated comprehensively: Information entropy is calculated based on the probability distribution of the fusion prediction results to reflect the degree of discrimination ambiguity of the current sample itself; The statistical variance is calculated based on the degree of fluctuation in the results of multiple forward inferences to reflect the uncertainty of the model's understanding of the sample. Inconsistency measures are calculated based on the distribution differences between the independent prediction results of each modality to reflect the degree of conflict between different modalities at the high-level semantic level. When the inconsistency metric is lower than a preset divergence threshold, an external knowledge consistency check is introduced: The category names in the independent prediction results of each modality are input into a pre-trained and parameter-frozen external text semantic encoder to obtain the predicted semantic representation. The raw input data of each modality is input to the external modality encoder respectively. The image input is to the external image encoder, the text input is to the external text encoder, and the speech is converted into text by the fixed speech recognition model and then input to the external text encoder to obtain the input semantic representation. Calculate the cosine similarity between each pair of predicted semantic representations and input semantic representations; If any cosine similarity is lower than the preset consistency threshold, the independent prediction of the modality is determined to lack semantic support. If all independent predictions of all modalities are deemed to lack semantic support, then the basic uncertainty will be forcibly set to a preset lower limit as an indicator of prediction uncertainty.

6. The AI ​​data accurate annotation method based on multimodal feature fusion as described in claim 5, characterized in that: The specific steps for performing confidence-weighted correction on the original labels based on prediction uncertainty to generate optimized labels are as follows: Set an uncertainty threshold; The prediction uncertainty index is compared with the uncertainty threshold, and the label softening coefficient is calculated based on the comparison result. The label softening coefficient is a non-negative number not exceeding 1, and increases with the increase of the prediction uncertainty index. The original hard-labeled labels are combined with the fused predicted probability distribution to generate optimized labels.

7. The AI ​​data accurate annotation method based on multimodal feature fusion as described in claim 6, characterized in that: Starting with the original input data, the process involves feature encoding, dynamic gating network processing, weighted fusion, cross-modal semantic alignment, and classification prediction to construct a labeled model. The model is then iteratively trained using optimized labeled tags. The specific steps are as follows: The multimodal input encoding module, the global context-guided dynamic gating module, the multi-granularity cross-modal alignment and fusion module, and the multi-source prediction and uncertainty quantification module are sequentially connected and parameterized into a neural network structure, wherein differentiable operations are used within each module and between modules, thereby forming an end-to-end labeled model. Based on the optimized label and the predicted probability distribution of the model output, calculate the cross-entropy loss; Update all learnable parameters of the labeled model using the backpropagation algorithm; Repeat the entire closed-loop process from acquiring raw input data to optimizing label generation and updating model parameters until the loss converges, thus completing the self-evolutionary iterative training.

8. A multimodal feature fusion-based AI data accurate annotation system, based on the multimodal feature fusion-based AI data accurate annotation method according to any one of claims 1 to 7, characterized in that: It includes an input encoding module, a gating weight module, a feature weighting module, a multi-granularity fusion module, an uncertainty calculation module, a label optimization module, and an iterative training module; The input encoding module acquires the original input data and performs feature encoding on each modality to obtain initial features; The gating weight module calculates global context features based on initial features, and inputs the global context features and initial features into the dynamic gating network to generate modality fusion weights. The feature weighting module weights the initial features according to the modality fusion weights to obtain weighted modal features; The multi-granularity fusion module extracts local fine-grained representations and global semantic representations from the weighted modal features, performs cross-modal fine-grained alignment based on the local fine-grained representations, brings the local fine-grained representations of semantic matching closer, and constrains the overall semantic consistency between different modalities through the global semantic representation to generate multi-granularity fusion features. The uncertainty calculation module performs category prediction based on multi-granularity fusion features, outputs the category prediction result, and calculates the prediction uncertainty. The label optimization module performs confidence-weighted correction on the original labels based on the prediction uncertainty to generate optimized labels. The iterative training module starts with the original input data and performs feature encoding, dynamic gating network processing, weighted fusion, cross-modal semantic alignment and classification prediction to form a labeled model. The module then uses optimized labels to iteratively train the labeled model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the AI ​​data accurate annotation method of multimodal feature fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the AI ​​data accurate annotation method of multimodal feature fusion as described in any one of claims 1 to 7.

Images