Intelligent labeling method and system based on multiple modes

Abstract

The invention discloses an intelligent labeling method and system based on multiple modes, and relates to the technical field of data labeling, and the method comprises the steps: collecting multi-mode data from multiple sources, preprocessing the collected data, labeling the multi-mode data according to task requirements, and dividing a labeled data set into a training set, a verification set and a test set; selecting a model architecture, setting training parameters, training the model by using training set data, and optimizing the model parameters; regularly evaluating model performance through a verification set, and adjusting a training strategy; evaluating the performance of the model on the test set, and improving the model according to a test result. According to the method, multi-modal large model fine adjustment is applied to labeling, an existing manual labeling mode is replaced, the labor cost of an enterprise is greatly saved, intelligent labeling can be used for coarse screening before data preparation, secondary verification can also be conducted on a small model result, the generalization ability of the model is improved, and a better labeling effect is brought.

Description

Technical Field

[0001] The present invention relates to the technical field of data annotation, and in particular to a multimodal-based intelligent annotation method and system. Background Art

[0002] With the popularization of devices such as the Internet of Things, social media, and surveillance cameras, the generation speed of multimedia data such as images and videos is growing exponentially. These massive data require efficient and accurate annotation for subsequent analysis and mining. Traditional manual annotation methods require a lot of human resources, are costly and inefficient. Especially in the case of huge amounts of data, the unsustainability of manual annotation becomes increasingly prominent. Manual annotation is easily affected by the subjective factors of the annotator, resulting in inconsistency in the annotation results. This inconsistency will directly affect the accuracy and reliability of data analysis. Multimodal data includes multiple types such as images, videos, and texts. The correlation and complementarity between them provide rich information for data analysis. However, how to effectively integrate and utilize this information has become a problem that needs to be solved urgently.

[0003] In image, video and text annotation, the current annotation method is usually manual. Whether it is data preparation before model training or data verification after model training, each file must be manually reviewed. However, this method is costly, subjective, easy to confuse, and inefficient. The traditional manual annotation method requires a lot of human resources and is costly. Manual annotation is easily affected by the subjective factors of the annotator, resulting in inconsistency in the annotation results. The annotation efficiency is low by manually annotating and reviewing each file. This paper designs a multimodal intelligent annotation method and system, which can effectively improve the efficiency of manual annotation. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal-based intelligent annotation method and system to solve the problems raised in the above background technology.

[0005] A multimodal intelligent labeling method, the intelligent labeling method comprising:

[0006] Collect multimodal data from multiple sources, preprocess the collected data, annotate the multimodal data according to task requirements, and divide the annotated data set into training set, validation set, and test set;

[0007] Select the model architecture, set the training parameters, train the model using the training set data, and optimize the model parameters;

[0008] Regularly evaluate model performance through validation sets and adjust training strategies;

[0009] Evaluate the performance of the model on the test set and improve the model based on the test results.

[0010] Preferably, the collecting multimodal data from multiple sources, preprocessing the collected data, labeling the multimodal data according to task requirements, and dividing the labeled data set into a training set, a validation set, and a test set specifically includes:

[0011] Obtain multimodal data based on the data API interface and save the collected data into the database;

[0012] Remove extra spaces, punctuation marks, and special characters in the text, including: #, @, $, %, ^, &, *, (), [], {}, |, \, :, ;, ", ', <, >, / , ~, perform word segmentation and remove stop words, crop, scale, and denoise images, decode video files, extract video frames, and unify text encoding, image format, and video format;

[0013] Develop detailed annotation standards based on specific tasks, including annotation types and label definitions, where the annotation types include classification, bounding box, and semantic segmentation;

[0014] Use annotation tools to annotate multimodal data;

[0015] Use random sampling to divide the labeled data into 70% training set, 20% validation set, and 10% test set;

[0016] Review the annotation results to check the accuracy and consistency of the annotations.

[0017] Preferably, the selecting a model architecture, setting training parameters, training the model using training set data, and optimizing the model parameters specifically include:

[0018] Use CNN to extract image features, use the embedding layer to convert text into vectors, use multiple layers of convolution and pooling operations to extract features, and connect the extracted features to the fully connected layer;

[0019] Concatenate image features and text features to form a unified feature representation, and select the appropriate output layer according to the task type;

[0020] Use the cross entropy loss function to set training parameters, where the cross entropy loss function expression is:

[0021]

[0022] In the formula, is the true label y and the predicted value The gap between them, N is the total number of samples, y i is the true label of the i-th sample, is the predicted value of the i-th sample;

[0023] Select Adam optimizer, set initial learning rate, and adjust learning rate decay;

[0024] Use the training set data for training, input images and text into the model, extract features through convolutional layers and fully connected layers, and perform feature fusion;

[0025] Calculate the loss value based on the model output and the true label, and use the back-propagation algorithm to calculate the gradient of the loss function to the model parameters;

[0026] Use the optimizer to update the parameters and continue forward propagation, loss calculation, backpropagation, and parameter update until the preset number of training rounds is reached.

[0027] Preferably, the step of updating parameters using an optimizer, continuing to perform forward propagation, loss calculation, back propagation, and parameter updating until a preset number of training rounds is reached specifically includes:

[0028] In the process of optimizing model parameters, the formula for updating parameters is:

[0029]

[0030] In the formula, θ represents the parameters of the model, is the learning rate, is the gradient of the loss function L with respect to the parameter θ.

[0031] Preferably, the step of regularly evaluating the model performance through the validation set and adjusting the training strategy specifically includes:

[0032] Use the training set to train the model according to the predetermined number of training rounds and batch size;

[0033] At the end of each batch, the model is evaluated using the validation set, and the performance indicators on the validation set are calculated, including the F1 score, precision, and recall, and the performance indicators of each batch are saved;

[0034] Compare the validation set performance of the current batch with the previous results to see whether the model is overfitting or underfitting;

[0035] If overfitting is found, set the patience parameter, stop training when the validation loss does not improve within multiple batches, perform regularization, and increase the amount of data by transforming the training data to improve the generalization ability of the model;

[0036] If underfitting is found, use more parameters, use a larger learning rate, increase the number of training rounds, retrain the model according to the adjusted strategy, and continue validation evaluation until the set performance standard is achieved.

[0037] Preferably, at the end of each batch, the model is evaluated using the validation set, the F1 score on the validation set is calculated, and the performance indicators of each batch are saved, specifically including:

[0038] At the end of each batch, the currently trained model is used to predict the validation set and obtain the predicted label;

[0039] Calculate the confusion matrix and calculate the values ​​in the confusion matrix based on the predicted labels and the true labels;

[0040] The calculation formulas for precision and recall are:

[0041]

[0042] Where TP is the number of samples correctly classified as positive, FP is the number of negative samples misclassified as positive, FN is the number of positive samples misclassified as negative, P is the proportion of all samples predicted to be positive that are truly positive, and R is the proportion of all samples that are actually positive that are correctly identified.

[0043] Based on the precision and recall, the F1 score is calculated, where the F1 score calculation formula is:

[0044]

[0045] Where F1 is the harmonic mean of precision and recall.

[0046] Preferably, evaluating the performance of the model on the test set and improving the model according to the test results specifically includes:

[0047] Use the trained model to predict the test set, obtain the predicted labels, calculate the performance indicators based on the predicted labels and the true labels, including the F1 score, precision and recall, and save the performance indicators of each batch

[0048] Analyze the above metrics, identify the model's performance on the test set, check the confusion matrix, and calculate the ratio of false positives to false negatives;

[0049] According to the analysis results, the model is improved, which includes adding new features, deleting irrelevant features, and performing feature extraction.

[0050] A multimodal intelligent labeling system, the intelligent labeling system comprising:

[0051] Dataset construction module: The dataset construction module is used to collect multimodal data and pre-process the collected data;

[0052] Data annotation module: The data annotation module is electrically connected to the data set construction module, and the data annotation module is used to annotate the multimodal data;

[0053] Model training module: The model training module is electrically connected to the data annotation module, and the model training module is used to train a multimodal large model;

[0054] Model verification module: The model verification module is electrically connected to the model training module, and is used to regularly evaluate the model performance on the verification set and adjust the training strategy in a timely manner;

[0055] Model testing module: The model testing module is electrically connected to the model verification module, and the model testing module is used to evaluate the performance of the model on the test set;

[0056] Model optimization module: The model optimization module is electrically connected to the model testing module, and the model optimization module is used to improve the model according to the test results to improve the performance and accuracy of the model.

[0057] The data set construction module specifically includes:

[0058] Data acquisition unit: acquires multimodal data based on the data API interface and saves the collected data into the database;

[0059] Data preprocessing unit: remove extra spaces, punctuation marks, and special characters in the text, perform word segmentation and remove stop words, crop, scale, and denoise images, decode video files, extract video frames, and unify text encoding, image format, and video format;

[0060] Data annotation definition unit: formulates detailed annotation standards according to specific tasks, including annotation types and label definitions, and the annotation types include classification, bounding box, and semantic segmentation.

[0061] The model verification module specifically includes:

[0062] Feature extraction unit: Use CNN to extract image features, use the embedding layer to convert text into vectors, use multi-layer convolution and pooling operations to extract features, and connect the extracted features to the fully connected layer;

[0063] Feature unification unit: concatenates image features and text features to form a unified feature representation, and selects the appropriate output layer according to the task type;

[0064] Parameter training unit: set training parameters using cross entropy loss function;

[0065] Parameter optimization unit: Select the Adam optimizer, set the initial learning rate, adjust the learning rate decay, use the optimizer to update the parameters, and continue to perform forward propagation, loss calculation, back propagation, and parameter update until the preset number of training rounds is reached.

[0066] By adopting the above technical solution, intelligent labeling of data is achieved.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] The present invention creatively constructs a multi-modal large model fine-tuning for labeling, replacing the existing manual labeling method, greatly saving the labor cost of enterprises;

[0069] The present invention will be applicable to the annotation of various modal data, such as images, videos and texts. The intelligent annotation of the present invention can be used for rough screening before data preparation and for secondary verification of small model results.

[0070] The multimodal large model fine-tuning of the present invention only requires a small amount of labeled data to improve the generalization ability of the model and greatly improve the labeling effect;

[0071] The present invention can include different business scenarios in a large multimodal model, and the model training cost can be greatly saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a step flow framework diagram of the present invention;

[0073] Figure 2 It is a system framework diagram of the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] See also Figure 1-2The present invention provides a technical solution: a multimodal intelligent labeling method, characterized in that the intelligent labeling method includes: collecting multimodal data from multiple sources, preprocessing the collected data, labeling the multimodal data according to task requirements, and dividing the labeled data set into a training set, a validation set, and a test set; selecting a model architecture, setting training parameters, using the training set data to train the model, and optimizing the model parameters; regularly evaluating the model performance through the validation set and adjusting the training strategy; evaluating the performance of the model on the test set, and improving the model according to the test results.

[0076] The collecting of multimodal data from multiple sources, preprocessing of the collected data, labeling of the multimodal data according to task requirements, and dividing the labeled data set into a training set, a validation set, and a test set specifically include:

[0077] Automatically collect multimodal data from multiple reliable data sources, including text, images, audio, and video, through a data-based API interface to ensure that all types of information required for the task are covered. Once the data is collected, it is systematically saved to the database.

[0078] Preprocess the collected text data, including removing extra spaces, punctuation marks and special characters. Special characters include: #, @, $, %, ^, &, *, (), [], {}, |, \, :, ;, ", ', <, >, / , ~;

[0079] Segment the text and remove stop words to retain keywords that are meaningful for analysis;

[0080] Crop, scale, and denoise image data to ensure image clarity and consistency, thereby improving the training effect of subsequent models;

[0081] Decode video files and extract video frames, unify video formats and frame rates, and ensure that all text encodings, image formats, and video formats are consistent;

[0082] Assign category labels to data, mark the location of objects in the image, draw bounding boxes, classify each pixel in the image to achieve refined annotation, and use annotation tools to annotate multimodal data;

[0083] The labeled data set is divided into 70% training set, 20% validation set and 10% test set by random sampling method, and the performance is evaluated on different data sets;

[0084] Conduct a comprehensive review of the annotation results to check the accuracy and consistency of the annotations.

[0085] The selecting of the model architecture, setting of the training parameters, training of the model using the training set data, and optimization of the model parameters specifically include:

[0086] Use CNN to extract image features, use the embedding layer to convert text into vectors, use multiple layers of convolution and pooling operations to extract features, and connect the extracted features to the fully connected layer;

[0087] Concatenate image features and text features to form a unified feature representation, and select the appropriate output layer according to the task type;

[0088] Use the cross entropy loss function to set training parameters, where the cross entropy loss function expression is:

[0089]

[0090] In the formula, is the true label y and the predicted value The gap between them, N is the total number of samples, y i is the true label of the i-th sample, is the predicted value of the i-th sample;

[0091] Select Adam optimizer, set initial learning rate, and adjust learning rate decay;

[0092] Use the training set data for training, input images and text into the model, extract features through convolutional layers and fully connected layers, and perform feature fusion;

[0093] Calculate the loss value based on the model output and the true label, and use the back-propagation algorithm to calculate the gradient of the loss function to the model parameters;

[0094] Use the optimizer to update the parameters and continue forward propagation, loss calculation, backpropagation, and parameter update until the preset number of training rounds is reached.

[0095] The model performance is regularly evaluated through the validation set, and the training strategy is adjusted, specifically including:

[0096] Use the training set to train the model according to the predetermined number of training rounds and batch size;

[0097] At the end of each batch, the model is evaluated using the validation set, and the performance indicators on the validation set are calculated, including the F1 score, precision, and recall, and the performance indicators of each batch are saved;

[0098] Compare the validation set performance of the current batch with the previous results to see whether the model is overfitting or underfitting;

[0099] If overfitting is found, set the patience parameter, stop training when the validation loss does not improve within multiple batches, perform regularization, and increase the amount of data by transforming the training data to improve the generalization ability of the model;

[0100] If underfitting is found, use more parameters, use a larger learning rate, increase the number of training rounds, retrain the model according to the adjusted strategy, and continue to perform validation evaluation until the set performance standard is achieved;

[0101] Among them, the F1 score is an indicator that comprehensively considers precision and recall. Precision is the ratio of the number of correctly predicted positive samples to the number of all predicted positive samples, and recall is the ratio of the number of correctly predicted positive samples to the number of all true positive samples. It measures the model's ability to capture positive samples. The performance indicators of each batch will be recorded and saved;

[0102] After each training cycle, compare the validation set performance of the current batch with the previous results to observe whether the model is overfitting or underfitting. If the validation set performance starts to decrease and the training set performance continues to increase, this usually indicates that the model may be overfitting;

[0103] If signs of overfitting are detected, measures are taken, including:

[0104] Set a patience parameter to automatically stop training when the validation loss does not improve over multiple consecutive batches;

[0105] Introducing regularization techniques to suppress the complexity of the model and encourage it to focus more on important features rather than noise;

[0106] By diversifying the training data, the number of training samples can be effectively increased.

[0107] At the end of each batch, the model is evaluated using the validation set, the F1 score on the validation set is calculated, and the performance indicators of each batch are saved, including:

[0108] At the end of each batch, the currently trained model is used to predict the validation set and obtain the predicted label;

[0109] Calculate the confusion matrix and calculate the values ​​in the confusion matrix based on the predicted labels and the true labels;

[0110] The calculation formulas for precision and recall are:

[0111]

[0112] Where TP is the number of samples correctly classified as positive, FP is the number of negative samples misclassified as positive, FN is the number of positive samples misclassified as negative, P is the proportion of all samples predicted to be positive that are truly positive, and R is the proportion of all samples that are actually positive that are correctly identified.

[0113] Based on the precision and recall, the F1 score is calculated, where the F1 score calculation formula is:

[0114]

[0115] Where F1 is the harmonic mean of precision and recall.

[0116] The performance of the model is evaluated on the test set, and the model is improved according to the test results, specifically including:

[0117] Use the trained model to predict the test set, obtain the predicted labels, calculate the performance indicators based on the predicted labels and the true labels, including the F1 score, precision and recall, and save the performance indicators of each batch

[0118] Analyze the above metrics, identify the model's performance on the test set, check the confusion matrix, and calculate the ratio of false positives to false negatives;

[0119] According to the analysis results, the model is improved.

[0120] Among them, model improvements include:

[0121] Introduce new features based on the results of data analysis;

[0122] Through feature selection techniques, features that do not contribute to model performance are identified and removed to reduce model complexity;

[0123] Feature extraction technology is used to transform the original features into a more meaningful feature space, thereby improving the learning efficiency and prediction ability of the model.

[0124] A multimodal intelligent labeling system, characterized in that the intelligent labeling system comprises:

[0125] Dataset construction module: The dataset construction module is used to collect multimodal data and pre-process the collected data;

[0126] Data annotation module: The data annotation module is electrically connected to the data set construction module, and the data annotation module is used to annotate the multimodal data;

[0127] Model training module: The model training module is electrically connected to the data annotation module, and the model training module is used to train a multimodal large model;

[0128] Model verification module: The model verification module is electrically connected to the model training module, and is used to regularly evaluate the model performance on the verification set and adjust the training strategy in a timely manner;

[0129] Model testing module: The model testing module is electrically connected to the model verification module, and the model testing module is used to evaluate the performance of the model on the test set;

[0130] Model optimization module: The model optimization module is electrically connected to the model testing module, and the model optimization module is used to improve the model according to the test results to improve the performance and accuracy of the model.

[0131] The data set construction module specifically includes:

[0132] Data acquisition unit: acquires multimodal data based on the data API interface and saves the collected data into the database;

[0133] Data preprocessing unit: remove extra spaces, punctuation marks, and special characters in the text, perform word segmentation and remove stop words, crop, scale, and denoise images, decode video files, extract video frames, and unify text encoding, image format, and video format;

[0134] Data annotation definition unit: formulates detailed annotation standards according to specific tasks, including annotation types and label definitions, and the annotation types include classification, bounding box, and semantic segmentation.

[0135] The model verification module specifically includes:

[0136] Feature extraction unit: Use CNN to extract image features, use the embedding layer to convert text into vectors, use multi-layer convolution and pooling operations to extract features, and connect the extracted features to the fully connected layer;

[0137] Feature unification unit: concatenates image features and text features to form a unified feature representation, and selects the appropriate output layer according to the task type;

[0138] Parameter training unit: set training parameters using cross entropy loss function;

[0139] Parameter optimization unit: Select the Adam optimizer, set the initial learning rate, adjust the learning rate decay, use the optimizer to update the parameters, and continue to perform forward propagation, loss calculation, back propagation, and parameter update until the preset number of training rounds is reached.

[0140] The use process of the present invention is:

[0141] Step 1: Collect multimodal data from multiple sources through API interfaces;

[0142] Step 2: Save the collected data into the database to ensure data security and accessibility;

[0143] Step 3: Remove extra spaces, punctuation marks and special characters from the text data;

[0144] Step 4: Crop, scale and denoise the image;

[0145] Step 5: Decode the video file and extract the video frames into a unified format;

[0146] Step 6: Perform word segmentation and remove stop words to ensure the clarity of text data;

[0147] Step 7: Define detailed annotation standards according to specific tasks, including annotation types;

[0148] Step 8: Use annotation tools to annotate multimodal data to ensure annotation quality;

[0149] Step 9: Use random sampling to divide the labeled data into 70% training set, 20% validation set and 10% test set;

[0150] Step 10: Check the accuracy and consistency of annotations to ensure data quality;

[0151] Step 11: Select the CNN model architecture for feature extraction;

[0152] Step 12: Use the embedding layer to convert the text data into a vector representation;

[0153] Step 13: Extract image features through convolution and pooling operations, and connect the features to the fully connected layer;

[0154] Step 14: Concatenate image features and text features to form a unified feature representation;

[0155] Step 15: Set training parameters using the cross entropy loss function;

[0156] Step 16: Select the Adam optimizer, set the initial learning rate and adjust the learning rate decay;

[0157] Step 17: Use the training set data for training and continuously update the model parameters through forward propagation and back propagation;

[0158] Step 18: At the end of each training batch, use the validation set to evaluate the model performance and calculate the F1 score, precision, and recall.

[0159] Step 19: Based on the performance of the validation set, observe whether the model is overfitting or underfitting;

[0160] Step 20: According to the evaluation results, set the patience parameter for regularization or adjust the learning rate and the number of training rounds;

[0161] Step 21: Evaluate the model performance on the test set, calculate the final performance indicators, and analyze the results;

[0162] Step 22: Based on the test results, identify false positives and false negatives and optimize the model, including adding new features or deleting irrelevant features.

[0163] In summary, the advantages of the present invention are:

[0164] By collecting multimodal data from multiple sources, the richness and diversity of data are enhanced. This integration can help the model better understand and process complex information and improve the accuracy and effectiveness of annotation.

[0165] The preprocessing steps include removing extra spaces, punctuation and special characters, image cropping, scaling and denoising, video frame extraction, etc. Such systematic processing ensures the consistency and quality of the data and provides a good foundation for subsequent annotation and training.

[0166] Detailed annotation standards are formulated according to specific tasks, including different annotation types, which can be flexibly adjusted according to actual needs, effectively improving the accuracy and applicability of annotations;

[0167] Regularly evaluate model performance through validation sets, observe overfitting and underfitting, and adjust training strategies in a timely manner to enhance the adaptability and robustness of the model;

[0168] In model training and verification, not only the F1 score is used, but also the precision and recall are combined to provide a comprehensive performance evaluation, making the performance of the model in real application scenarios more valuable for reference;

[0169] Model improvements are made based on test results, including feature addition, deletion of irrelevant features, and feature extraction, forming a closed-loop feedback mechanism to ensure continuous model optimization;

[0170] The system design modularizes each functional module, which enhances the maintainability and scalability of the system and is conducive to future technology upgrades and functional expansions.

[0171] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A multimodal intelligent annotation method, characterized in that: Smart labeling methods include: Collect multimodal data from multiple sources, preprocess the collected data, annotate the multimodal data according to task requirements, and divide the annotated data set into training set, validation set, and test set; Select the model architecture, set the training parameters, train the model using the training set data, and optimize the model parameters; Regularly evaluate model performance through validation sets and adjust training strategies; Evaluate the performance of the model on the test set and improve the model based on the test results.

2. The multimodal intelligent annotation method according to claim 1, characterized in that: The collecting of multimodal data from multiple sources, preprocessing of the collected data, labeling of the multimodal data according to task requirements, and dividing the labeled data set into a training set, a validation set, and a test set specifically include: Obtain multimodal data based on the data API interface and save the collected data into the database; Remove extra spaces, punctuation marks, and special characters from text, perform word segmentation and remove stop words, crop, scale, and denoise images, decode video files, extract video frames, and unify text encoding, image format, and video format; Develop detailed annotation standards based on specific tasks, including annotation types and label definitions, where the annotation types include classification, bounding box, and semantic segmentation; Use annotation tools to annotate multimodal data; Use random sampling to divide the labeled data into 70% training set, 20% validation set, and 10% test set; Review the annotation results to check the accuracy and consistency of the annotations.

3. The multimodal intelligent annotation method according to claim 2, characterized in that: The selecting of the model architecture, setting of the training parameters, training of the model using the training set data, and optimization of the model parameters specifically include: Use CNN to extract image features, use the embedding layer to convert text into vectors, use multiple layers of convolution and pooling operations to extract features, and connect the extracted features to the fully connected layer; Concatenate image features and text features to form a unified feature representation, and select the appropriate output layer according to the task type; Use the cross entropy loss function to set training parameters, where the cross entropy loss function expression is: In the formula, is the true label y and the predicted value The gap between them, N is the total number of samples, y i is the true label of the i-th sample, is the predicted value of the i-th sample; Select Adam optimizer, set initial learning rate, and adjust learning rate decay; Use the training set data for training, input images and text into the model, extract features through convolutional layers and fully connected layers, and perform feature fusion; Calculate the loss value based on the model output and the true label, and use the back-propagation algorithm to calculate the gradient of the loss function to the model parameters; Use the optimizer to update the parameters and continue forward propagation, loss calculation, backpropagation, and parameter update until the preset number of training rounds is reached.

4. The multimodal intelligent annotation method according to claim 3, characterized in that: The method of using the optimizer to update parameters, continuing forward propagation, loss calculation, back propagation, and parameter update until a preset number of training rounds is reached specifically includes: In the process of optimizing model parameters, the formula for updating parameters is: In the formula, θ represents the parameters of the model, is the learning rate, is the gradient of the loss function L with respect to the parameter θ.

5. The multimodal intelligent annotation method according to claim 4, characterized in that: The model performance is regularly evaluated through the validation set, and the training strategy is adjusted specifically including: Use the training set to train the model according to the predetermined number of training rounds and batch size; At the end of each batch, the model is evaluated using the validation set, and the performance indicators on the validation set are calculated, including the F1 score, precision, and recall, and the performance indicators of each batch are saved; Compare the validation set performance of the current batch with the previous results to see whether the model is overfitting or underfitting; If overfitting is found, set the patience parameter, stop training when the validation loss does not improve within multiple batches, perform regularization, and increase the amount of data by transforming the training data to improve the generalization ability of the model; If underfitting is found, use more parameters, use a larger learning rate, increase the number of training rounds, retrain the model according to the adjusted strategy, and continue validation evaluation until the set performance standard is achieved.

6. The multimodal intelligent annotation method according to claim 5, characterized in that: At the end of each batch, the model is evaluated using the validation set, the F1 score on the validation set is calculated, and the performance indicators of each batch are saved, including: At the end of each batch, the currently trained model is used to predict the validation set and obtain the predicted label; Calculate the confusion matrix and calculate the values ​​in the confusion matrix based on the predicted labels and the true labels; The calculation formulas for precision and recall are: Where TP is the number of samples correctly classified as positive, FP is the number of negative samples misclassified as positive, FN is the number of positive samples misclassified as negative, P is the proportion of all samples predicted to be positive that are truly positive, and R is the proportion of all samples that are actually positive that are correctly identified. Based on the precision and recall, the F1 score is calculated, where the F1 score calculation formula is: Where F1 is the harmonic mean of precision and recall.

7. The multimodal intelligent annotation method according to claim 6, characterized in that: The performance of the model is evaluated on the test set, and the model is improved according to the test results, specifically including: Use the trained model to predict the test set, obtain the predicted labels, calculate the performance indicators based on the predicted labels and the true labels, including the F1 score, precision and recall, and save the performance indicators of each batch Analyze the above metrics, identify the model's performance on the test set, check the confusion matrix, and calculate the ratio of false positives to false negatives; According to the analysis results, the model is improved, which includes adding new features, deleting irrelevant features, and performing feature extraction.

8. A multimodal intelligent annotation system, characterized in that: The intelligent labeling system comprises: Dataset construction module: The dataset construction module is used to collect multimodal data and pre-process the collected data; Data annotation module: The data annotation module is electrically connected to the data set construction module, and the data annotation module is used to annotate the multimodal data; Model training module: The model training module is electrically connected to the data annotation module, and the model training module is used to train a multimodal large model; Model verification module: The model verification module is electrically connected to the model training module, and is used to regularly evaluate the model performance on the verification set and adjust the training strategy in a timely manner; Model testing module: The model testing module is electrically connected to the model verification module, and the model testing module is used to evaluate the performance of the model on the test set; Model optimization module: The model optimization module is electrically connected to the model testing module, and the model optimization module is used to improve the model according to the test results to improve the performance and accuracy of the model.

9. The multimodal intelligent annotation system according to claim 8, characterized in that: The data set construction module specifically includes: Data acquisition unit: acquires multimodal data based on the data API interface and saves the collected data into the database; Data preprocessing unit: remove extra spaces, punctuation marks, and special characters in the text, perform word segmentation and remove stop words, crop, scale, and denoise images, decode video files, extract video frames, and unify text encoding, image format, and video format; Data annotation definition unit: formulates detailed annotation standards according to specific tasks, including annotation types and label definitions, and the annotation types include classification, bounding box, and semantic segmentation.

10. The multimodal intelligent annotation system according to claim 9, characterized in that: The model verification module specifically includes: Feature extraction unit: Use CNN to extract image features, use the embedding layer to convert text into vectors, use multi-layer convolution and pooling operations to extract features, and connect the extracted features to the fully connected layer; Feature unification unit: concatenates image features and text features to form a unified feature representation, and selects the appropriate output layer according to the task type; Parameter training unit: set training parameters using cross entropy loss function; Parameter optimization unit: Select the Adam optimizer, set the initial learning rate, adjust the learning rate decay, use the optimizer to update the parameters, and continue to perform forward propagation, loss calculation, back propagation, and parameter update until the preset number of training rounds is reached.

Images