Intelligent data annotation method and system based on multi-modal large model

Abstract

The invention relates to the technical field of data processing, and discloses an intelligent data labeling method and system based on a multi-modal large model. The method comprises the following steps: receiving text, image and video data to generate a multi-modal embedding vector, carrying out annotation prediction and outputting a confidence score, extracting scene features from a low-confidence sample, clustering to generate a scene label, calculating gradient sensitivity of each network layer to construct a scene specificity parameter layer set, and carrying out parameter updating by adopting a hierarchical learning rate. And when the accuracy of the general scene is reduced and exceeds a threshold value, returning parameters and reducing the learning rate for re-updating. According to the method and the device, the accuracy of multi-modal data labeling in a complex scene and the adaptive capability of model parameter optimization are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an intelligent data annotation method and system based on a multimodal large model. Background Technology

[0002] The development of artificial intelligence technology relies heavily on high-quality, large-scale, and diverse labeled data. Existing multimodal data labeling methods mainly employ pre-trained large models to automatically label text, images, videos, and other data. After the encoder extracts modal features, the decoder generates the labeled results, significantly improving efficiency compared to purely manual labeling. These methods typically use a uniform labeling process for all data. After the model outputs the labeled results, manual sampling and correction are performed. The corrected data can be used for incremental learning or fine-tuning of the model to gradually improve labeling performance.

[0003] However, existing technologies have the following shortcomings: First, the model's annotation accuracy drops significantly when facing specific challenging scenarios, such as low-light, heavily occluded scenes, and densely distributed long-tail categories. Current methods lack an automatic identification mechanism for these scenarios leading to inaccurate annotations, relying instead on pre-defined scene classification rules based on human experience, which is insufficient to cover the diverse and challenging scenarios encountered in real-world applications. Second, existing model fine-tuning methods use a uniform learning rate for parameter updates across all network layers, failing to identify which layers have a greater impact on annotation performance in specific challenging scenarios. This results in a lack of targeted parameter optimization, making the model prone to overfitting when adapting to specific scenarios, thus impairing its generalization ability in regular scenarios. Summary of the Invention

[0004] This application provides an intelligent data annotation method and system based on multimodal large models, which solves the problems of existing multimodal data annotation methods being unable to automatically identify difficult scenarios, lacking targeted parameter optimization, and having difficulty balancing the adaptability of specific scenarios and the generalization of general scenarios during model fine-tuning. It improves the accuracy of multimodal data annotation in complex scenarios and the adaptive capability of model parameter optimization.

[0005] Firstly, this application provides an intelligent data annotation method based on a multimodal large model, the intelligent data annotation method based on a multimodal large model comprising:

[0006] Step S1: Receive text data, image data, and video data, and generate multimodal embedding vectors after modal encoding processing;

[0007] Step S2: Label and predict the multimodal embedding vectors and output confidence scores. Construct a low-confidence label set from samples whose confidence scores are lower than a preset threshold.

[0008] Step S3: Extract the illumination condition features, occlusion degree features and target density features of each sample in the low confidence label set, generate scene labels through clustering, calculate the gradient sensitivity of each network layer on the samples corresponding to the scene labels, and construct a set of scene-specific parameter layers from the top 30% of the network layers ranked by gradient sensitivity.

[0009] Step S4: Set a first learning rate for the parameter layers in the scene-specific parameter layer set and a second learning rate for the remaining parameter layers to update the parameters. The first learning rate is 50 times the second learning rate. When the annotation accuracy on the general scene validation set decreases by more than the fallback threshold after the parameter update, fall back the parameters in the scene-specific parameter layer set and reduce the first learning rate to update again.

[0010] Secondly, this application provides an intelligent data annotation system based on a multimodal large model, the intelligent data annotation system based on a multimodal large model comprising:

[0011] The generation module receives text data, image data, and video data, and generates multimodal embedding vectors after modal encoding processing.

[0012] The prediction module is used to annotate and predict the multimodal embedding vectors and output a confidence score, and to construct a low-confidence annotation set from samples whose confidence scores are lower than a preset threshold.

[0013] The clustering module is used to extract the illumination condition features, occlusion degree features and target density features of each sample in the low confidence label set, generate scene labels through clustering, calculate the gradient sensitivity of each network layer on the sample corresponding to the scene label, and construct a set of scene-specific parameter layers from the top 30% of the network layers ranked by gradient sensitivity.

[0014] The update module is used to set a first learning rate for the parameter layers in the scene-specific parameter layer set and a second learning rate for the remaining parameter layers to update the parameters. The first learning rate is 50 times the second learning rate. When the annotation accuracy on the general scene validation set decreases by more than the fallback threshold after the parameter update, the parameters in the scene-specific parameter layer set are fallbacked and the first learning rate is reduced and the parameters are updated again.

[0015] The technical solution provided in this application achieves homogenized representation of heterogeneous data by receiving text, image, and video data and generating unified multimodal embedding vectors through modal encoding, thus eliminating differences in format and standards between different modal data. By synchronously outputting confidence scores during the annotation and prediction process, this invention can quantitatively assess the predictive certainty of the model for each sample. A low-confidence annotation set is constructed from samples with confidence scores below a preset threshold, enabling automatic filtering of areas of annotation uncertainty. Compared to existing technologies that rely on manual sampling to identify annotation errors, this invention proactively identifies samples with significant uncertainty in model predictions, avoiding the randomness and bias of manual sampling. By extracting the illumination condition features, occlusion degree features, and target density features of each sample in the low-confidence annotation set, the technical feature characterizes the scene attributes leading to inaccurate annotations from a content perspective. Scene labels are generated through clustering, achieving data-driven automatic discovery of difficult scenes without the need for manually preset scene classification rules, thus adapting to diverse and challenging scenes encountered in practical applications. The technical feature of this invention is to calculate the gradient sensitivity of each network layer on the samples corresponding to the scene labels and construct a set of scene-specific parameter layers from the top 30% of the network layers ranked by gradient sensitivity. This technique identifies the network layers that have the most significant impact on the labeling performance of a specific scene through quantitative analysis, providing a precise basis for parameter localization for subsequent layered optimization. Compared with the existing technology that adopts a uniform optimization strategy for all network layers, the layered identification mechanism of this invention makes parameter optimization more targeted. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an embodiment of the intelligent data annotation method based on a multimodal large model in this application.

[0018] Figure 2 This is a schematic diagram illustrating the gradient sensitivity distribution of each network layer in different scenarios in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram comparing the impact of different learning rate strategies on annotation performance in the embodiments of this application. Detailed Implementation

[0020] This application provides an intelligent data annotation method and system based on a multimodal large model. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the intelligent data annotation method based on multimodal large models in this application includes:

[0022] Step S1: Receive text data, image data, and video data, and generate multimodal embedding vectors after modal encoding processing;

[0023] Specifically, this invention addresses multimodal data annotation scenarios. In step S1, the received text data includes unstructured text such as documents and dialogue records; image data includes static visual content such as surveillance photos and medical images; and video data includes dynamic visual sequences such as dashcam recordings and security monitoring. Modal encoding processing transforms heterogeneous data into a unified vector representation. Text uses a pre-trained language model to extract semantic embeddings, images use a visual encoder to extract visual features, and videos first extract keyframes and then perform feature aggregation. Ultimately, all three modalities are transformed into fixed-dimensional embedding vectors, achieving homogeneous representation of heterogeneous data.

[0024] Step S2: Label and predict the multimodal embedding vectors and output confidence scores. Construct a low-confidence label set from samples whose confidence scores are lower than a preset threshold.

[0025] Specifically, in step S2, the annotation prediction process inputs multimodal embedding vectors into a deep neural network. After the network extracts semantic features through the encoder, the decoder generates corresponding outputs based on the specific annotation task. The object detection task outputs bounding box coordinates and categories, the entity recognition task outputs label sequences, and the action recognition task outputs action categories and timestamps. The confidence score quantifies the model's certainty about the prediction results and is calculated by recording the probability distribution during the generation process. Samples with confidence scores below a preset threshold indicate that the model has significant uncertainty in labeling the data. These samples are selected to construct a low-confidence annotation set, providing target data for subsequent scene analysis and model optimization.

[0026] Step S3: Extract the illumination condition features, occlusion degree features and target density features of each sample in the low confidence label set, generate scene labels through clustering, calculate the gradient sensitivity of each network layer on the samples corresponding to the scene labels, and construct a set of scene-specific parameter layers from the top 30% of the network layers ranked by gradient sensitivity.

[0027] Specifically, illumination features determine the shooting environment by analyzing image brightness distribution; occlusion features calculate the occlusion ratio of targets using segmentation algorithms; and target density features reflect the density of target distribution in the image. After concatenating these features into a scene feature vector, a density clustering algorithm automatically discovers clustered regions of similar scenes based on the sample density distribution in the feature space, without the need for manual pre-setting of scene categories. Gradient sensitivity analysis calculates the gradient magnitude of the parameters of each network layer with respect to the loss function for a specific scene through backpropagation. Layers with large gradient magnitudes indicate that changes in their parameters have a significant impact on the scene annotation performance. The top 30% of layers in gradient sensitivity ranking are selected to construct a set of scene-specific parameter layers, which will be the focus of optimization in subsequent fine-tuning.

[0028] Step S4: Set a first learning rate for the parameter layers in the scene-specific parameter layer set and a second learning rate for the remaining parameter layers to update the parameters. The first learning rate is 50 times the second learning rate. When the annotation accuracy on the general scene validation set decreases by more than the fallback threshold after the parameter update, fall back the parameters in the scene-specific parameter layer set and reduce the first learning rate to update again.

[0029] Specifically, a hierarchical learning rate strategy is adopted. A larger first learning rate is set for the scene-specific parameter layer to enable it to quickly adapt to the specific scene, while a very small second learning rate is set for the other parameter layers to maintain their general feature extraction capabilities. The first learning rate is 50 times that of the second learning rate to achieve differentiated optimization intensity. Parameter updates adopt a batch accumulation mechanism, accumulating sufficient samples before triggering updates to smooth gradient noise. Immediately after the update, the labeled accuracy is tested on a general scene validation set. If the accuracy drops by more than the backoff threshold, it indicates that the model is overfitting the specific scene and impairing generalization ability. The parameter backoff mechanism is then triggered to restore the scene-specific layer parameters to their pre-update state. At the same time, the first learning rate is reduced and the update is repeated. By dynamically adjusting the learning rate, a balance is achieved between the adaptability of the specific scene and the generalization of the general scene.

[0030] In one specific embodiment, step S1 includes:

[0031] The text data is converted into an encoding format and segmented by sentence boundaries to obtain segmented text data. The segmented text data is then input into a pre-trained encoder to extract semantic features and obtain text embedding vectors.

[0032] The image data is normalized to obtain normalized image data. The normalized image data is then segmented into a sequence of image blocks and input into a visual encoder to extract visual features, thereby obtaining an image embedding vector.

[0033] A scene switching detection algorithm is used to extract key frames from video data to obtain a key frame sequence. Each frame in the key frame sequence is visually encoded and the inter-frame similarity is calculated to generate a weighted coefficient. Based on the weighted coefficient, a weighted average aggregation is performed to obtain a video embedding vector.

[0034] The text embedding vector, the image embedding vector, and the video embedding vector are uniformly constructed into a multimodal embedding vector.

[0035] Specifically, text data encoding format conversion unifies text from different sources to the UTF-8 encoding standard, eliminating garbled characters caused by character set differences. Sentence boundary segmentation processing divides long texts into independent semantic units by recognizing punctuation marks such as periods, question marks, and exclamation marks, with each segment's length controlled within 512 tokens to meet the input limit of the pre-trained encoder. The pre-trained encoder employs a multi-layer Transformer structure, capturing contextual dependencies between words through a self-attention mechanism, extracting feature representations containing semantic information, and using the output vector at a specially marked position as the semantically condensed representation of the entire text, i.e., the text embedding vector. Image data size normalization scales images of different resolutions to a fixed size, maintaining aspect ratio and avoiding target distortion. Image patch segmentation divides the entire image into fixed-size patches, each patch being converted into a vector representation through linear projection. The visual encoder performs global feature extraction on the image patch sequence, and the output after multi-layer encoding is compressed into a single vector, i.e., the image embedding vector, through global average pooling.

[0036] The scene transition detection algorithm for video data detects shot transition points by calculating optical flow changes between adjacent frames. It uniformly samples and extracts representative keyframes in each continuous scene segment to avoid computational waste caused by redundant frames. Each frame in the keyframe sequence has its features extracted by a visual encoder to obtain a frame embedding vector sequence. Frame similarity is measured by calculating the cosine distance between the frame embedding vectors to assess the content relevance of adjacent frames. Frames with high similarity are assigned higher weights, and frames with low similarity are assigned lower weights. After normalizing the weighting coefficients, the frame embedding vector sequences are weighted and averaged to enhance the features of similar frames while suppressing the features of abruptly changing frames. The aggregated vector is the video embedding vector. The three modal embedding vectors have the same vector dimension and can be directly concatenated or integrated into a unified multimodal embedding vector through a fusion layer, achieving collaborative expression of cross-modal information.

[0037] In one specific embodiment, step S2, which involves labeling and predicting the multimodal embedding vector and outputting a confidence score, includes:

[0038] The multimodal embedding vector is input into the encoder to perform multi-head self-attention calculation to extract semantic features, thereby obtaining the encoded feature vector;

[0039] The encoded feature vector is input into the decoder, which uses an autoregressive method to generate annotation results step by step, thus obtaining a predicted annotation sequence.

[0040] During the generation of the predicted labeled sequence, the generation probability of each labeled element is recorded. For the target detection task, the product of the maximum probability value of the category and the bounding box confidence is recorded. For the sequence labeling task, the joint probability of the label sequence is recorded to obtain the confidence score.

[0041] A low-confidence label set is constructed from samples whose confidence scores are lower than a preset threshold and their corresponding multimodal embedding vectors.

[0042] Specifically, after the multimodal embedding vector is input into the encoder, a multi-head self-attention mechanism divides the input vector into multiple attention heads for parallel processing. Each attention head independently calculates three matrices: query, key, and value. The attention weight is obtained through the dot product of the query and key, representing the correlation strength between features at different locations. The weighted sum of the vectors is then concatenated with the outputs of the multiple attention heads and subjected to a linear transformation to obtain the encoded feature vector, which incorporates global semantic information. After the encoded feature vector is input into the decoder, the decoder adopts an autoregressive generation method. That is, each time a labeled element is generated, the previously generated elements are used as conditions. The decoder gradually predicts the next element through its self-attention layer and cross-attention layer. The object detection task gradually generates bounding box coordinates and category labels, the sequence labeling task gradually generates BIO tags for each token, and the action recognition task gradually generates action categories and timestamps, ultimately forming a complete predicted and labeled sequence.

[0043] During the generation process, the decoder outputs the probability distribution of all candidate labels at each time step and records the probability value of the actual generated label. For object detection, the maximum softmax probability predicted by the category is multiplied by the location confidence score output by the bounding box regression module to obtain the overall confidence score. For sequence labeling, the probability of each position in the entire label sequence is multiplied together to obtain the joint sequence probability as the confidence score. For action recognition, the product of the maximum softmax probability of the action category and the temporal consistency score is taken as the confidence score. A preset threshold is set to 0.80. Samples with confidence scores below this threshold are considered to have high model prediction uncertainty. These samples, along with their corresponding multimodal embedding vectors, are saved to construct a low-confidence label set. This dataset contains the original feature representations and erroneous prediction results of the inaccurately labeled samples.

[0044] In one specific embodiment, step S3 extracts the illumination condition features, occlusion degree features, and target density features of each sample in the low-confidence annotation set, including:

[0045] The peak positions of the brightness histogram are calculated from the image data to obtain the illumination condition characteristics;

[0046] The occlusion degree feature is obtained by calculating the ratio of occluded target pixels to total target pixels using a segmentation algorithm.

[0047] The ratio of the number of detected targets in an image to the image area is calculated to obtain the target density feature;

[0048] The lighting condition features, the occlusion degree features, and the target density features are concatenated to construct a scene feature vector.

[0049] Specifically, the brightness histogram of the image data statistically analyzes the grayscale value distribution of each pixel. The horizontal axis represents the grayscale levels from 0 to 255, and the vertical axis represents the number of pixels at the corresponding grayscale level. The peak position is the grayscale level with the most pixels. A peak value less than 50 indicates an overly dark environment, between 50 and 200 indicates normal lighting, and greater than 200 indicates an overly bright environment. This peak value serves as the lighting condition feature. The occlusion degree feature uses an instance segmentation algorithm to annotate targets in the image at the pixel level, distinguishing between foreground targets and background areas. For partially occluded targets, the algorithm identifies the visible and occluded parts, calculates the occlusion ratio by dividing the number of occluded pixels by the total number of pixels for that target, and takes the average of the occlusion ratios for all targets as the occlusion degree feature for the entire image.

[0050] The target density feature is obtained by counting the number of all detected targets in the image using a target detection algorithm. This number is then divided by the total pixel area of ​​the image to obtain the target distribution density per unit area. A higher density value indicates a denser target distribution in the image, while a lower density value indicates a sparser target distribution. The illumination condition feature, occlusion feature, and target density feature are sequentially arranged and concatenated into a one-dimensional vector. This vector contains three numerical elements, each corresponding to a quantified description of a scene attribute. The concatenated vector is the scene feature vector, which encapsulates the key scene information affecting the accuracy of annotation.

[0051] In one specific embodiment, step S3 involves generating scene labels through clustering, calculating the gradient sensitivity of each network layer on the samples corresponding to the scene labels, and constructing a set of scene-specific parameter layers from the top 30% of the network layers ranked by gradient sensitivity, including:

[0052] The scene feature vectors of all samples in the low-confidence annotation set are clustered using a density clustering algorithm to obtain multiple scene clusters;

[0053] For each scene cluster, calculate the feature mean vector, identify the feature dimension that deviates from the global mean by more than a set multiple of the standard deviation as the dominant feature, and generate scene labels based on the dominant feature;

[0054] Samples are extracted in groups according to the scene labels. Each sample is input into the network for forward propagation to calculate the prediction loss. During backpropagation, the gradient norm of each network layer parameter is recorded to obtain the gradient sensitivity of each network layer.

[0055] The gradient sensitivity is sorted from high to low, and the top 30% of network layers are selected to construct a set of scene-specific parameter layers.

[0056] Specifically, the density clustering algorithm DBSCAN measures sample similarity by calculating the Euclidean distance between scene feature vectors. Samples with a distance less than 0.35 of the neighborhood radius are considered neighbors, and samples containing at least 8 neighbors are defined as core points. Starting from the core point, the samples in its neighborhood are recursively expanded to form a scene cluster. Samples that do not belong to the neighborhood of any core point are marked as noise points. The algorithm automatically identifies density-continuous regions as independent scene clusters without pre-specifying the number of clusters. After clustering, multiple scene clusters are obtained.

[0057] The feature mean vector of each cluster is obtained by averaging the scene feature vectors of all samples within each scene cluster along each dimension. Simultaneously, the global feature mean and standard deviation of each dimension for all samples in the low-confidence label set are calculated. The cluster mean vector is compared dimension-by-dimensionally with the global mean vector. When the cluster mean in a certain dimension deviates from the global mean by more than 1.5 times the standard deviation, that dimension is identified as the dominant feature. For example, the mean of the lighting condition feature of a certain cluster is 45, while the global mean is 120 and the standard deviation is 30. The deviation is 75, exceeding 1.5 times the standard deviation of 45, making lighting condition the dominant feature of that cluster. Similarly, the mean of the occlusion degree feature of that cluster is 0.42, while the global mean is 0.15 and the standard deviation is 0.12. The deviation is 0.27, exceeding 1.5 times the standard deviation of 0.18, making occlusion degree also a dominant feature. Based on the combination of dominant features, the scene label "low light, heavy occlusion" is generated.

[0058] The samples in the low-confidence annotation set are grouped according to scene labels. Each group contains all samples with the same scene label. Samples from a certain group and their corresponding ground truth labels are extracted. The multimodal embedding vectors of the samples are input into a neural network for forward propagation. The network contains 30 layers, including an 18-layer encoder and a 12-layer decoder. The encoder extracts features layer by layer through multi-head self-attention and a feedforward neural network. The decoder generates predicted labels based on the encoded features. The loss function between the predicted labels and the ground truth labels is calculated. The loss for the object detection task includes a weighted sum of classification cross-entropy loss, bounding box regression loss, and intersection-union ratio loss. The loss for the sequence labeling task is the negative log-likelihood loss.

[0059] During backpropagation, the partial derivative of the loss function with respect to the parameters of each layer is used to obtain the gradient matrix. The Frobenius norm of the gradient matrix of each layer, which is the square root of the sum of squares of all elements, is calculated as the influence strength of the parameter changes of that layer on the loss. The gradient norm of all samples in the group is taken as a weighted average to obtain the gradient sensitivity of that layer in that scenario. The weighting coefficient is the sample labeling error, and the larger the error, the higher the weight.

[0060] Repeat the above process for all scene clusters to obtain the gradient sensitivity of each layer in each scene. The comprehensive gradient sensitivity score of each layer is obtained by weighted summation according to the frequency of scene occurrence. The scores of the 30 layers are sorted from high to low. The top 30%, i.e. the top 9 layers, are selected to construct a set of scene-specific parameter layers. These layers have the most significant impact on the annotation performance of uncertain scenes. The layers in the set are mainly located in the deep layers of the encoder and the shallow layers of the decoder, and are responsible for high-order semantic extraction and annotation mapping transformation.

[0061] Figure 2 This is a schematic diagram illustrating the gradient sensitivity distribution of each network layer in different scenarios in the embodiments of this application. For example... Figure 2 As shown, the horizontal axis represents the 30 layers (L1 to L30) of the neural network, and the vertical axis represents the gradient sensitivity values ​​of the parameters in each layer. The figure contains three curves, corresponding to low-light scenes (solid lines marked with circles), heavily occluded scenes (dashed lines marked with squares), and high-density scenes (dotted lines marked with triangles). The dashed horizontal line marks the threshold position of the top 30% of gradient sensitivity, and the gray shaded area above the threshold line represents the distribution range of scene-specific parameter layers. It can be observed from the figure that the gradient sensitivity distribution curves for the three scenes show similar trends: the gradient sensitivity of the early layers (L1-L6) is low, indicating that the shallow network mainly extracts general low-level features; the gradient sensitivity of the middle layers (L7-L13) rises rapidly and reaches a peak, with the sensitivity of layers L11-L13 all exceeding 0.85, indicating that these layers have the most significant impact on the annotation performance of specific scenes; the gradient sensitivity of the deep layers (L14-L30) gradually decreases. Based on the top 30% threshold selection, layers L8-L16 (9 layers in total) were identified as scene-specific parameter layers (corresponding to the shaded area in the figure). These layers will be given a larger learning rate in subsequent parameter updates to quickly adapt to specific scenes, while the remaining 21 layers will maintain a smaller learning rate to preserve general feature extraction capabilities. This figure provides a quantitative basis for the selection of scene-specific parameter layers.

[0062] In one specific embodiment, step S4, which involves setting a first learning rate for the parameter layers in the scene-specific parameter layer set and setting a second learning rate for the remaining parameter layers, includes:

[0063] A refined dataset is constructed by extracting manually reviewed and corrected samples from the low-confidence labeled set;

[0064] Set a cumulative batch threshold, and trigger a parameter update once the number of fine-calibrated samples reaches the cumulative batch threshold.

[0065] A first learning rate is set for the parameter layers in the scene-specific parameter layer set, and a second learning rate is set for the remaining parameter layers. The first learning rate is 50 times the second learning rate. An optimizer is used to update the parameters, and a snapshot of the model parameters after the parameter update is saved.

[0066] Specifically, samples in the low-confidence annotation set need to be reviewed by human experts. For object detection tasks, experts check whether the bounding boxes accurately select targets, whether the category labels are correct, and whether there are any missed targets. For entity recognition tasks, they check whether the entity boundaries are complete, whether the entity types are accurate, and whether nested entities are correctly identified. For action recognition tasks, they check whether the action categories are accurate, whether the start and end timestamps are precise, and whether compound actions are fully annotated. Experts correct incorrectly predicted annotations by pairing the corrected accurate annotations with the original multimodal embedding vectors, and also annotate the specific reasons for the errors, such as targets being occluded by the foreground leading to missed detections, or polysemous words being misinterpreted as having a general meaning in a professional context. Samples corrected by experts constitute the refined annotation dataset, which contains accurate annotation results and detailed error attribution information. The cumulative batch threshold is set to 64. Samples in the refined annotation dataset are accumulated sequentially over time. When the accumulated number of samples reaches 64, a parameter update operation is triggered. The batch accumulation mechanism avoids excessive perturbation of model parameters by annotation noise from a single sample or special cases. By averaging the gradient information of multiple samples, the parameter update direction becomes more stable and reliable.

[0067] The scene-specific parameter layer set comprises the aforementioned 9 network layers for identification. A first learning rate of 0.00008 is set for these 9 layers, and a second learning rate of 0.0000016 is set for the remaining 21 general parameter layers. The first learning rate is 50 times that of the second. This differential setting allows the parameters of the scene-specific layers to quickly adapt to the annotation patterns of specific scenes in the precision-calibrated dataset, while the parameters of the general parameter layers remain almost frozen, undergoing only minor adjustments to maintain cross-scene feature extraction capabilities. The optimizer employs the AdamW algorithm, which maintains first-moment and second-moment estimates for each parameter. The first-moment estimate is an exponentially weighted moving average of historical gradients, and the second-moment estimate is an exponentially weighted moving average of the squares of historical gradients. Momentum coefficients are set to 0.9 and 0.999, respectively, and a weight decay coefficient is set to 0.01. L2 regularization constraints are applied to the parameters to prevent overfitting. When updating parameters, AdamW calculates the adaptive step size for each parameter based on gradient and moment estimation. Scene-specific layers have larger step sizes due to their larger learning rates, while general layers have smaller step sizes due to their smaller learning rates. After 15 iterations, the parameter weights of all 30 layers and the moment estimation state of the optimizer are saved as a snapshot of the model parameters. The snapshot file records complete information such as the current batch number, parameter values ​​of each layer, and optimizer state variables, which facilitates the restoration of the model state during subsequent performance verification or parameter rollback.

[0068] In one specific embodiment, when the decrease in annotation accuracy on the general scene validation set after parameter update in step S4 exceeds the backoff threshold, the parameters in the scene-specific parameter layer set are rolled back and the first learning rate is reduced before being updated again, including:

[0069] A general scenario validation set is constructed by randomly selecting samples from historical high-confidence labeled data that meet the criteria of confidence scores higher than a set value and scene labels that do not belong to uncertain scenarios.

[0070] The model with updated input parameters of the general scenario validation set is labeled and predicted, and the labeling accuracy is calculated to obtain the updated accuracy.

[0071] Load the model parameter snapshot before the parameter update, calculate the labeled accuracy on the general scenario validation set, and obtain the accuracy before the update;

[0072] Calculate the percentage of the difference between the updated accuracy and the original accuracy relative to the original accuracy. When the negative value of the percentage exceeds the rollback threshold, roll back the parameters in the scene-specific parameter layer set to the original values, reduce the first learning rate, and then re-execute the parameter update.

[0073] Specifically, the historical high-confidence annotation database stores samples with confidence scores higher than 0.90 from past annotation tasks. These samples represent common scene data where the model's predictions were highly certain before parameter updates. From this database, 300 samples are randomly selected, requiring a confidence score higher than 0.90 and the scene label not belonging to the aforementioned uncertain scene clusters. This excludes specific scenes such as low-light heavy occlusion and densely distributed long-tail categories, retaining only general scene samples from steady-state operation to construct a general scene validation set. The validation set is then input into the updated model for annotation and prediction. For object detection, the mean average accuracy (mAP) is calculated; for sequence labeling, the F1 score is calculated; and for action recognition, the Top-1 accuracy is calculated. The updated accuracy is obtained by weighting the multi-task validation set according to sample size.

[0074] Load the saved snapshot of the model parameters before the parameter update, restore the parameter weights and optimizer states of all 30 layers, input the same general scenario validation set into the snapshot model for labeled prediction and calculate the accuracy to obtain the accuracy before the update. The difference between the updated accuracy and the original accuracy is calculated. This difference is then divided by the original accuracy and multiplied by 100 to obtain the percentage change in accuracy. A positive percentage indicates performance improvement, while a negative percentage indicates performance degradation. The rollback threshold is set to -2.0%. When the percentage is less than -2.0%, meaning the general scene accuracy drops by more than 2%, it is determined that the model is overfitting to a specific scene and impairing its generalization ability. This triggers a parameter rollback mechanism, restoring the parameter values ​​of the 9th layer in the scene-specific parameter layer set to the values ​​of the corresponding layers in the pre-update snapshot. At the same time, the current parameters of the 21st general parameter layer are retained because their changes are minor. After the rollback, the first learning rate is reduced from 0.00008 to 0.00004, which is multiplied by a decay factor of 0.5. After reducing the learning rate, the parameter update process is re-executed, but the number of iterations is reduced to 8, and an early stopping mechanism is added. If the loss on the validation set does not decrease for 3 consecutive iterations, training is terminated early. If the performance constraints are still not met after 3 rollbacks and retraining, automatic fine-tuning is paused, and the current batch of data is marked as a difficult batch and pushed to human experts for in-depth analysis.

[0075] Figure 3 This is a schematic diagram comparing the impact of different learning rate strategies on annotation performance in the embodiments of this application. Figure 3 As shown, the horizontal axis represents four different learning rate strategies: a unified learning rate baseline method, double the learning rate (10 times that of the general layer for scene-specific layers), the method of this invention (50 times), and an excessively large learning rate (100 times); the vertical axis represents the percentage change in accuracy. The upper diagonal-filled bars in the figure represent the improvement in accuracy for specific scenes, while the lower reverse diagonal-filled bars represent the decrease in accuracy for general scenes. As shown in the figure, while the uniform learning rate strategy has the least impact on general scenarios (only a 0.5% decrease), its improvement in specific scenarios is also the most limited (only 8.5%), resulting in a net improvement of 8.0%. The double learning rate strategy achieves a 15.2% improvement in specific scenarios, but a 1.2% decrease in general scenarios, resulting in a net improvement of 14.0%. The 50x learning rate strategy adopted in this invention achieves the largest improvement of 23.8% in specific scenarios. Although it decreases by 2.1% in general scenarios, the net improvement reaches 21.7%, significantly outperforming other strategies. Conversely, the excessively high 100x learning rate, while improving by 19.5% in specific scenarios, leads to a significant 5.8% decrease in general scenarios, resulting in a net improvement of only 13.7%, indicating that over-optimization can damage the model's generalization ability. This figure verifies that the 50x learning rate strategy of this invention achieves the best balance between adaptability in specific scenarios and generalization in general scenarios.

[0076] The above describes the intelligent data annotation method based on a multimodal large model in the embodiments of this application. The following describes the intelligent data annotation system based on a multimodal large model in the embodiments of this application. One embodiment of the intelligent data annotation system based on a multimodal large model in the embodiments of this application includes:

[0077] The generation module receives text data, image data, and video data, and generates multimodal embedding vectors after modal encoding processing.

[0078] The prediction module is used to annotate and predict the multimodal embedding vectors and output a confidence score, and to construct a low-confidence annotation set from samples whose confidence scores are lower than a preset threshold.

[0079] The clustering module is used to extract the illumination condition features, occlusion degree features and target density features of each sample in the low confidence label set, generate scene labels through clustering, calculate the gradient sensitivity of each network layer on the sample corresponding to the scene label, and construct a set of scene-specific parameter layers from the top 30% of the network layers ranked by gradient sensitivity.

[0080] The update module is used to set a first learning rate for the parameter layers in the scene-specific parameter layer set and a second learning rate for the remaining parameter layers to update the parameters. The first learning rate is 50 times the second learning rate. When the annotation accuracy on the general scene validation set decreases by more than the fallback threshold after the parameter update, the parameters in the scene-specific parameter layer set are fallbacked and the first learning rate is reduced and the parameters are updated again.

[0081] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal large model-based intelligent data labeling method, characterized in that, The method includes: Step S1: Receive text data, image data, and video data, and generate multimodal embedding vectors after modal encoding processing; Step S2: Label and predict the multimodal embedding vectors and output confidence scores. Construct a low-confidence label set from samples whose confidence scores are lower than a preset threshold. Step S3: Extract the illumination condition features, occlusion degree features and target density features of each sample in the low confidence label set, generate scene labels through clustering, calculate the gradient sensitivity of each network layer on the samples corresponding to the scene labels, and construct a set of scene-specific parameter layers from the top 30% of the network layers ranked by gradient sensitivity. Step S4: Set a first learning rate for the parameter layers in the scene-specific parameter layer set and a second learning rate for the remaining parameter layers to update the parameters. The first learning rate is 50 times the second learning rate. When the annotation accuracy on the general scene validation set decreases by more than the fallback threshold after the parameter update, fall back the parameters in the scene-specific parameter layer set and reduce the first learning rate to update again.

2. The intelligent data labeling method based on a multi-modal large model according to claim 1, characterized in that, Step S1 includes: The text data is converted into an encoding format and segmented by sentence boundaries to obtain segmented text data. The segmented text data is then input into a pre-trained encoder to extract semantic features and obtain text embedding vectors. The image data is normalized to obtain normalized image data. The normalized image data is then segmented into a sequence of image blocks and input into a visual encoder to extract visual features, thereby obtaining an image embedding vector. A scene switching detection algorithm is used to extract key frames from video data to obtain a key frame sequence. Each frame in the key frame sequence is visually encoded and the inter-frame similarity is calculated to generate a weighted coefficient. Based on the weighted coefficient, a weighted average aggregation is performed to obtain a video embedding vector. The text embedding vector, the image embedding vector, and the video embedding vector are uniformly constructed into a multimodal embedding vector. 3.The intelligent data labeling method based on a multi-modal large model according to claim 1, wherein, Step S2, which involves labeling and predicting the multimodal embedding vectors and outputting confidence scores, includes: The multimodal embedding vector is input into the encoder to perform multi-head self-attention calculation to extract semantic features, thereby obtaining the encoded feature vector; The encoded feature vector is input into the decoder, which uses an autoregressive method to generate annotation results step by step, thus obtaining a predicted annotation sequence. During the generation of the predicted labeled sequence, the generation probability of each labeled element is recorded. For the target detection task, the product of the maximum probability value of the category and the bounding box confidence is recorded. For the sequence labeling task, the joint probability of the label sequence is recorded to obtain the confidence score. A low-confidence label set is constructed from samples whose confidence scores are lower than a preset threshold and their corresponding multimodal embedding vectors. 4.The intelligent data labeling method based on a multi-modal large model according to claim 1, wherein, Step S3 extracts the illumination condition features, occlusion degree features, and target density features of each sample in the low-confidence annotation set, including: The peak positions of the brightness histogram are calculated from the image data to obtain the illumination condition characteristics; The occlusion degree feature is obtained by calculating the ratio of occluded target pixels to total target pixels using a segmentation algorithm. The ratio of the number of detected targets in an image to the image area is calculated to obtain the target density feature; The lighting condition features, the occlusion degree features, and the target density features are concatenated to construct a scene feature vector.

5. The intelligent data annotation method based on a multimodal large model according to claim 4, characterized in that, In step S3, scene labels are generated through clustering, and the gradient sensitivity of each network layer on the samples corresponding to the scene labels is calculated. A set of scene-specific parameter layers is constructed from the top 30% of the network layers ranked by gradient sensitivity, including: The scene feature vectors of all samples in the low-confidence annotation set are clustered using a density clustering algorithm to obtain multiple scene clusters; For each scene cluster, calculate the feature mean vector, identify the feature dimension that deviates from the global mean by more than a set multiple of the standard deviation as the dominant feature, and generate scene labels based on the dominant feature; Samples are extracted in groups according to the scene labels. Each sample is input into the network for forward propagation to calculate the prediction loss. During backpropagation, the gradient norm of each network layer parameter is recorded to obtain the gradient sensitivity of each network layer. The gradient sensitivity is sorted from high to low, and the top 30% of network layers are selected to construct a set of scene-specific parameter layers.

6. The intelligent data annotation method based on a multimodal large model according to claim 1, characterized in that, Step S4, which involves setting a first learning rate for the parameter layers in the scene-specific parameter layer set and setting a second learning rate for the remaining parameter layers, includes: A refined dataset is constructed by extracting manually reviewed and corrected samples from the low-confidence labeled set; Set a cumulative batch threshold, and trigger a parameter update once the number of fine-calibrated samples reaches the cumulative batch threshold. A first learning rate is set for the parameter layers in the set of scene-specific parameter layers, and a second learning rate is set for the remaining parameter layers. The first learning rate is 50 times the second learning rate. An optimizer is used to update the parameters, and a snapshot of the model parameters after the parameter update is saved.

7. The intelligent data annotation method based on a multimodal large model according to claim 6, characterized in that, In step S4, when the accuracy of the annotation on the general scenario validation set decreases beyond the fallback threshold after the parameter update, the parameters in the scenario-specific parameter layer set are rolled back and the first learning rate is reduced before being updated again. This includes: A general scenario validation set is constructed by randomly selecting samples from historical high-confidence labeled data that meet the criteria of confidence scores higher than a set value and scene labels that do not belong to uncertain scenarios. The model with updated input parameters of the general scenario validation set is labeled and predicted, and the labeling accuracy is calculated to obtain the updated accuracy. Load the model parameter snapshot before the parameter update, calculate the labeled accuracy on the general scenario validation set, and obtain the accuracy before the update; Calculate the percentage of the difference between the updated accuracy and the original accuracy relative to the original accuracy. When the negative value of the percentage exceeds the rollback threshold, roll back the parameters in the scene-specific parameter layer set to the original values, reduce the first learning rate, and then re-execute the parameter update.

8. An intelligent data annotation system based on a multimodal large model, characterized in that, For implementing the intelligent data annotation method based on a multimodal large model as described in any one of claims 1-7, the intelligent data annotation system based on a multimodal large model comprises: The generation module receives text data, image data, and video data, and generates multimodal embedding vectors after modal encoding processing. The prediction module is used to annotate and predict the multimodal embedding vectors and output a confidence score, and to construct a low-confidence annotation set from samples whose confidence scores are lower than a preset threshold. The clustering module is used to extract the illumination condition features, occlusion degree features and target density features of each sample in the low confidence label set, generate scene labels through clustering, calculate the gradient sensitivity of each network layer on the sample corresponding to the scene label, and construct a set of scene-specific parameter layers from the top 30% of the network layers ranked by gradient sensitivity. The update module is used to set a first learning rate for the parameter layers in the scene-specific parameter layer set and a second learning rate for the remaining parameter layers to update the parameters. The first learning rate is 50 times the second learning rate. When the annotation accuracy on the general scene validation set decreases by more than the fallback threshold after the parameter update, the parameters in the scene-specific parameter layer set are fallbacked and the first learning rate is reduced and the parameters are updated again.

9. The system according to claim 8, characterized in that, It receives text data, image data, and video data, and generates multimodal embedding vectors after modal encoding processing, including: The text data is converted into an encoding format and segmented by sentence boundaries to obtain segmented text data. The segmented text data is then input into a pre-trained encoder to extract semantic features and obtain text embedding vectors. The image data is normalized to obtain normalized image data. The normalized image data is then segmented into a sequence of image blocks and input into a visual encoder to extract visual features, thereby obtaining an image embedding vector. A scene switching detection algorithm is used to extract key frames from video data to obtain a key frame sequence. Each frame in the key frame sequence is visually encoded and the inter-frame similarity is calculated to generate a weighted coefficient. Based on the weighted coefficient, a weighted average aggregation is performed to obtain a video embedding vector. The text embedding vector, the image embedding vector, and the video embedding vector are uniformly constructed into a multimodal embedding vector.

10. The system according to claim 8, characterized in that, The multimodal embedding vectors are labeled, predicted, and confidence scores are output, including: The multimodal embedding vector is input into the encoder to perform multi-head self-attention calculation to extract semantic features, thereby obtaining the encoded feature vector; The encoded feature vector is input into the decoder, which uses an autoregressive method to generate annotation results step by step, thus obtaining a predicted annotation sequence. During the generation of the predicted labeled sequence, the generation probability of each labeled element is recorded. For the target detection task, the product of the maximum probability value of the category and the bounding box confidence is recorded. For the sequence labeling task, the joint probability of the label sequence is recorded to obtain the confidence score. A low-confidence label set is constructed from samples whose confidence scores are lower than a preset threshold and their corresponding multimodal embedding vectors.

Images