Multi-modal data labeling method and system, electronic equipment and storage medium
By analyzing and preprocessing multimodal data, calling adaptive annotation model and performing audit and fine-tuning, the problems of low efficiency and poor consistency of multimodal data annotation are solved, and efficient and accurate data annotation is achieved.
Patent Information
- Application Number
- CN202510746523.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-05
- Publication Date
- 2025-08-26
AI Technical Summary
In the prior art, multimodal data labeling is inefficient and inconsistent, and it is impossible to efficiently perform accurate labeling.
Provide a multimodal data annotation method. After obtaining the data to be marked, analyzing the annotation relationship according to the data type, pre-processing is performed, and the corresponding annotation model is called for analysis, and the audit process and fine-tuning interface are triggered when the confidence is lower than the threshold to ensure the accuracy of the annotation.
It realizes efficient and accurate labeling of multimodal data, adapts to various scenarios and data types, corrects errors in a timely manner, and improves the accuracy and consistency of the labeling model.
Smart Images
Figure CN120541527A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal data annotation method and system, electronic equipment, and storage medium. Background Art
[0002] A model is an abstract computing system constructed through mathematical structures, algorithms, and data training. It is used to simulate human intelligent behavior, solve specific problems, or achieve specific functions. It is the core carrier of AI technology and is essentially a tool for learning patterns from data and making inferences, predictions, or decisions. Therefore, models with different functions require training using different types of data and their annotations, necessitating the annotation of multimodal data.
[0003] In the field of artificial intelligence, large amounts of data and its annotations are required to train models. Therefore, with the rapid development of AI technology, the scale and complexity of multimodal data are growing exponentially. Due to the different data structures and characteristics of each modality, the current method of manually reviewing and annotating the data of each modality is still to be used.
[0004] Although manual labeling ensures accurate labeling of data adapted to each modality, it has the problems of low efficiency and inconsistent labeling. Summary of the Invention
[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a multimodal data annotation method and system, electronic device, and storage medium to solve the problem that the prior art cannot accurately and efficiently annotate data.
[0006] In order to achieve the above objectives, this application provides the following technical solutions:
[0007] The first aspect of the present application provides a multimodal data annotation method, comprising:
[0008] Get the current data to be labeled;
[0009] If the current data to be annotated is text data, importing the current data to be annotated;
[0010] If the current data to be annotated is non-text data, analyzing the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated, and importing the current data to be annotated and its annotation relationship data;
[0011] Preprocessing the data to be labeled using a corresponding preprocessing method according to the type of the data to be labeled;
[0012] According to the current scene type and the type of the current data to be annotated, calling the corresponding annotation model to analyze the preprocessed current data to be annotated, and outputting the annotation of the current data to be annotated and its confidence; wherein the current data to be annotated for non-text data is analyzed based on its annotation relationship data;
[0013] If the confidence level of the annotation of the current data to be annotated is lower than a preset threshold, a review process is triggered for the annotation of the current data to be annotated;
[0014] When a target event is monitored, a fine-tuning interface call is triggered to fine-tune the annotation model using the reviewed current data to be annotated and its annotations through the fine-tuning interface; wherein the target event refers to an event that updates the annotation of the data.
[0015] Optionally, in the above-mentioned multimodal data labeling method, analyzing the labeling relationship data of the current data to be labeled according to the type of the current data to be labeled includes:
[0016] If the current data to be annotated is image data, identifying a marking file that matches the current data to be annotated through an adapter;
[0017] If the current data to be labeled is video data, identifying a labeling file that matches the current data to be labeled;
[0018] Convert the format of the current data to be annotated and parse the matching annotation file;
[0019] Performing time axis mapping on the current data to be annotated after format conversion and the annotated file data obtained by parsing, to obtain the annotated file data that matches each frame image in the current data to be annotated;
[0020] If the current data to be labeled is video data, basic information extraction, multi-track processing, and attached data analysis operations are performed on the current data to be labeled.
[0021] Optionally, in the above-mentioned multimodal data annotation method, the current data to be annotated is text data, and the preprocessing of the current data to be annotated using a corresponding preprocessing method according to the type of the current data to be annotated includes:
[0022] Performing unified coding conversion on the current data to be annotated;
[0023] Performing structural standardization on the current data to be annotated after encoding conversion;
[0024] Adding corresponding language tags to the current data to be annotated after structural standardization;
[0025] The data to be annotated after the language tags are added is cleaned and quality checked.
[0026] Optionally, in the above-mentioned multimodal data labeling method, the current data to be labeled is image data, and the preprocessing of the current data to be labeled using a corresponding preprocessing method according to the type of the current data to be labeled includes:
[0027] Performing blur detection on each image to be labeled in the current data to be labeled;
[0028] Rechecking the local fuzziness of each of the to-be-annotated images whose fuzziness is greater than the fuzziness threshold using a multi-scale sliding window;
[0029] Removing the image to be labeled if the blurriness of any local area is lower than the blurriness threshold;
[0030] For each of the remaining images to be annotated, convert the image to be annotated into the HSV color space, extract the data of the brightness channel, and construct a brightness histogram;
[0031] Calculating the pixel ratio of each overexposed area and the pixel ratio of each underexposed area of the image to be labeled according to the brightness histogram;
[0032] Marking the image to be marked as having an exposure abnormality and removing it if any of the overexposed areas or the underexposed areas has a pixel ratio greater than a preset ratio;
[0033] Resizing and graying the remaining images to be labeled;
[0034] Extracting low-frequency components of each grayscale image to be labeled by discrete cosine transform, and generating fingerprint hash values of each image to be labeled by using the low-frequency components of each image to be labeled;
[0035] Calculating the distance between each two images to be annotated based on the fingerprint hash values of each two images to be annotated;
[0036] Deduplication is performed on the images to be labeled whose distance is less than a preset distance.
[0037] Optionally, in the above-mentioned multimodal data labeling method, the current data to be labeled is video data, and the preprocessing of the current data to be labeled using a corresponding preprocessing method according to the type of the current data to be labeled includes:
[0038] Decoding each video to be labeled in the current data to be labeled;
[0039] Filtering out low-quality videos to be labeled based on parameters of each video to be labeled;
[0040] Key frames are extracted from each of the filtered videos to be labeled.
[0041] Optionally, in the above-mentioned multimodal data labeling method, the current data to be labeled is audio data, and the preprocessing of the current data to be labeled using a corresponding preprocessing method according to the type of the current data to be labeled includes:
[0042] Converting the audio files to a unified format in the current data to be annotated;
[0043] Performing noise reduction and enhancement on each of the audios to be annotated after the format conversion;
[0044] Based at least on the decodability, effective duration, and sampling rate consistency of each of the audios to be labeled after noise reduction and enhancement, the audios to be labeled that do not meet the requirements are screened out and filtered out.
[0045] A second aspect of the present application provides a multimodal data annotation system, comprising:
[0046] A data acquisition unit, used to acquire the current data to be labeled;
[0047] A first importing unit, configured to import the current data to be annotated when the current data to be annotated is text data;
[0048] A second importing unit is configured to analyze the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated when the current data to be annotated is non-text data, and import the current data to be annotated and its annotation relationship data;
[0049] A preprocessing unit, configured to preprocess the current data to be labeled using a corresponding preprocessing method according to the type of the current data to be labeled;
[0050] a labeling unit, configured to analyze the preprocessed data to be labeled by calling a corresponding labeling model according to the current scene type and the type of the data to be labeled, and output the labeling of the data to be labeled and its confidence level; wherein the data to be labeled that is non-text data is analyzed based on its labeling relationship data;
[0051] An audit unit, configured to trigger an audit process for the annotation of the current data to be annotated when the confidence level of the annotation of the current data to be annotated is lower than a preset threshold;
[0052] A fine-tuning unit is used to trigger a fine-tuning interface call when a target event is monitored, so as to fine-tune the annotation model using the reviewed current data to be annotated and its annotations through the fine-tuning interface; wherein the target event refers to an event that updates the annotation of the data.
[0053] Optionally, in the above-mentioned multimodal data annotation system, when the second importing unit performs the step of analyzing the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated, it is configured to:
[0054] If the current data to be annotated is image data, identifying a marking file that matches the current data to be annotated through an adapter;
[0055] If the current data to be labeled is video data, identifying a labeling file that matches the current data to be labeled;
[0056] Convert the format of the current data to be annotated and parse the matching annotation file;
[0057] Performing time axis mapping on the current data to be annotated after format conversion and the annotated file data obtained by parsing, to obtain the annotated file data that matches each frame image in the current data to be annotated;
[0058] If the current data to be labeled is video data, basic information extraction, multi-track processing, and attached data analysis operations are performed on the current data to be labeled.
[0059] Optionally, in the above-mentioned multimodal data annotation system, the current data to be annotated is text data, and when the preprocessing unit performs the preprocessing of the current data to be annotated using a corresponding preprocessing method according to the type of the current data to be annotated, it is used to:
[0060] Performing unified coding conversion on the current data to be annotated;
[0061] Performing structural standardization on the current data to be annotated after encoding conversion;
[0062] Adding corresponding language tags to the current data to be annotated after structural standardization;
[0063] The data to be annotated after the language tags are added is cleaned and quality checked.
[0064] Optionally, in the above-mentioned multimodal data annotation system, the current data to be annotated is image data, and when the preprocessing unit performs the preprocessing of the current data to be annotated using a corresponding preprocessing method according to the type of the current data to be annotated, it is used to:
[0065] Performing blur detection on each image to be labeled in the current data to be labeled;
[0066] Rechecking the local fuzziness of each of the to-be-annotated images whose fuzziness is greater than the fuzziness threshold using a multi-scale sliding window;
[0067] Removing the image to be labeled if the blurriness of any local area is lower than the blurriness threshold;
[0068] For each of the remaining images to be annotated, convert the image to be annotated into the HSV color space, extract the data of the brightness channel, and construct a brightness histogram;
[0069] Calculating the pixel ratio of each overexposed area and the pixel ratio of each underexposed area of the image to be labeled according to the brightness histogram;
[0070] Marking the image to be marked as having an exposure abnormality and removing it if any of the overexposed areas or the underexposed areas has a pixel ratio greater than a preset ratio;
[0071] Resizing and graying the remaining images to be labeled;
[0072] Extracting low-frequency components of each grayscale image to be labeled by discrete cosine transform, and generating fingerprint hash values of each image to be labeled by using the low-frequency components of each image to be labeled;
[0073] Calculating the distance between each two images to be annotated based on the fingerprint hash values of each two images to be annotated;
[0074] Deduplication is performed on the images to be labeled whose distance is less than a preset distance.
[0075] Optionally, in the above-mentioned multimodal data labeling system, the current data to be labeled is video data, and when the preprocessing unit performs the preprocessing of the current data to be labeled using a corresponding preprocessing method according to the type of the current data to be labeled, it is used to:
[0076] Decoding each video to be labeled in the current data to be labeled;
[0077] Filtering out low-quality videos to be labeled based on parameters of each video to be labeled;
[0078] Key frames are extracted from each of the filtered videos to be labeled.
[0079] Optionally, in the above-mentioned multimodal data annotation system, the current data to be annotated is audio data, and when the preprocessing unit performs the preprocessing of the current data to be annotated using a corresponding preprocessing method according to the type of the current data to be annotated, it is used to:
[0080] Converting the audio files to a unified format in the current data to be annotated;
[0081] Performing noise reduction and enhancement on each of the audios to be annotated after the format conversion;
[0082] Based at least on the decodability, effective duration, and sampling rate consistency of each of the audios to be labeled after noise reduction and enhancement, the audios to be labeled that do not meet the requirements are screened out and filtered out.
[0083] A third aspect of the present application provides an electronic device, including:
[0084] memory and processor;
[0085] Wherein, the memory is used to store programs;
[0086] The processor is used to execute the program, and when the program is executed, it is specifically used to implement the multimodal data labeling method as described in any one of the above.
[0087] In a fourth aspect, the present application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the multimodal data labeling method as described in any one of the above.
[0088] The present application provides a multimodal data annotation method, which obtains current data to be annotated, which may include data of various modalities. If the current data to be annotated is text data, the current data to be annotated is imported. If the current data to be annotated is non-text data, the annotation relationship data of the current data to be annotated is analyzed according to the type of the current data to be annotated, and the current data to be annotated and its annotation relationship data are imported so that accurate annotation of the corresponding data can be achieved based on the annotation relationship data in the future. Then, according to the type of the current data to be annotated, the current data to be annotated is preprocessed using a corresponding preprocessing method to filter out problematic data, etc., and data that can be subsequently analyzed can be obtained. Then, according to the current scene type and the type of the current data to be annotated, the corresponding annotation model is called to analyze the preprocessed current data to be annotated, and the annotation and confidence of the current data to be annotated are output. Among them, the current data to be annotated for non-text data is analyzed based on its annotation relationship data. Thus, according to the scene type and the type of data, the corresponding annotation model is called to perform annotation prediction, so that it can adapt to various scene types and data types and ensure the accuracy of annotation. Furthermore, if the confidence level of the annotation of the current data to be annotated is lower than a preset threshold, the audit process for the annotation of the current data to be annotated is triggered, so that a small amount of erroneous data that may exist in the model annotation can be audited and promptly corrected. Finally, when the target event is monitored, the fine-tuning interface call is triggered to fine-tune the annotation model using the audited current data to be annotated and its annotations through the fine-tuning interface. Among them, the target event refers to the event that updates the annotation of the data, so that the adjusted data can be used to fine-tune the annotation model in a timely manner, thereby further ensuring the accuracy of the annotation model's annotation of the data, thereby realizing a method for efficiently and accurately annotating data. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.
[0090] Figure 1 A flowchart of a multimodal data annotation method provided in an embodiment of the present application;
[0091] Figure 2 A flowchart of a method for preprocessing text data provided in an embodiment of the present application;
[0092] Figure 3 A flowchart of a method for preprocessing image data provided in an embodiment of the present application;
[0093] Figure 4 A flowchart of a method for preprocessing video data provided in an embodiment of the present application;
[0094] Figure 5 A flowchart of a method for preprocessing audio data provided in an embodiment of the present application;
[0095] Figure 6 A schematic diagram of the architecture of a multimodal data annotation system provided in an embodiment of the present application;
[0096] Figure 7 A schematic diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0097] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0098] In this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.
[0099] The present application embodiment provides a multimodal data annotation method, such as Figure 1 As shown, the following steps are included:
[0100] S101: Obtain the current data to be labeled.
[0101] It should be noted that in the embodiment of the present application, the ability to import and unify abnormal data is provided, so the data that needs to be labeled for each modality can be imported, so the current data to be labeled is the data of any modality uploaded by the user.
[0102] S102: Determine whether the current data to be annotated is text data.
[0103] Since text data can be directly analyzed accurately, while graphs, videos, and audio formats are more complex and require further analysis, if the data to be labeled is text data, step S103 is executed. If the data to be labeled is non-text data, step S104 is executed.
[0104] S103: Import the current data to be labeled.
[0105] For text data, we support mainstream text formats such as TXT, CSV, JSONL, and CoNLL. Furthermore, our dynamic parsing engine can automatically identify file encoding and data structure. For API imports, we provide a RESTful interface for connecting to enterprise databases or third-party storage.
[0106] Alternatively, considering that large text files are not easy to upload directly, you can use fragmented upload and in-memory streaming technology to split the data into blocks by line or paragraph to avoid memory overflow. Specifically, you can combine it with a distributed framework to achieve parallel import.
[0107] S104 , analyzing the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated, and importing the current data to be annotated and its annotation relationship data.
[0108] Optionally, non-text data can be uploaded to a designated directory or a third-party storage system first, and the path or unique identifier can be recorded for easy use in subsequent processing.
[0109] Specifically, for the current non-text data to be annotated, in order to facilitate the subsequent accurate annotation of the current data to be annotated, it is necessary to analyze its annotation relationship data in a targeted manner according to its type, that is, to analyze the data related to the current data to be annotated, so as to facilitate the data that users rely on for annotation.
[0110] Optionally, in another embodiment of the present application, a specific implementation method of analyzing the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated includes:
[0111] If the current data to be annotated is image data, an adapter is used to identify a annotation file that matches the current data to be annotated.
[0112] Optionally, adapters such as VOC and COCO can automatically identify the mapping between XML or JSON annotation files and images, thereby obtaining annotation files that match the data to be annotated. For example, the VOC format requires strict correspondence between the JPEGImages and Annotations directories, so file name hashing can be used to achieve an accurate match. Furthermore, for folder-based imports, subdirectory names are automatically parsed as category labels.
[0113] If the current data to be annotated is video data, the annotation file that matches the current data to be annotated is identified, and the format of the current data to be annotated is converted and the annotation file that matches it is parsed. The current data to be annotated after format conversion and the annotation file data obtained by parsing are then time-mapped to obtain the annotation file data that matches each frame image in the current data to be annotated.
[0114] Alternatively, a tool such as FFmpeg can be used for format conversion and parsing, thereby implementing XML / JSON syntax tree parsing and timeline mapping. For annotated file matching, the mapping relationship between the video and the annotated file can also be matched using the file name hash.
[0115] If the current data to be labeled is video data, basic information extraction, multi-audio track processing and ancillary data analysis operations are performed on the current data to be labeled.
[0116] Optionally, for audio data, a parsing module can be built based on FFmpeg and Librosa libraries to perform operations such as basic information extraction, multi-track processing, and ancillary data parsing.
[0117] S105 : Preprocess the data to be labeled using a corresponding preprocessing method according to the type of the data to be labeled.
[0118] It should be noted that since the imported data to be annotated cannot be directly input into the model for analysis and may contain inconsistent formats and abnormal data, different types of data need to be preprocessed to facilitate subsequent annotation and improve annotation efficiency and accuracy. Preprocessing can optionally include data format conversion, data cleaning, and abnormal data filtering.
[0119] Optionally, in another embodiment of the present application, when the current data to be annotated is text data, a specific implementation of step S105 is as follows: Figure 2 Shown, including:
[0120] S201: Perform unified coding conversion on the current data to be labeled.
[0121] Optionally, you can convert the text data to UTF-8 encoding to solve the garbled code problem.
[0122] S202: Structural standardization is performed on the current data to be annotated after the encoding conversion.
[0123] Specifically, considering that there may be data with other structures such as tables, it is necessary to standardize the structure and convert the tables and other data into JSON arrays.
[0124] S203: Add corresponding language tags to the current data to be annotated after structural standardization.
[0125] It should be noted that, considering that the current data to be annotated may include texts in multiple languages, in order to facilitate subsequent analysis of the language of the text and to ensure the accuracy of the annotation, corresponding language tags are added to the current data to be annotated.
[0126] S204: Perform data cleaning and quality verification on the current data to be annotated after the language tags are added.
[0127] Optionally, specific data cleaning and quality verification may include deleting special characters, filtering sensitive information, deduplication, integrity verification, etc.
[0128] Optionally, in another embodiment of the present application, when the current data to be labeled is image data, a specific implementation of step S105 is as follows: Figure 3 Shown, including:
[0129] S301 , performing blur detection on each image to be labeled in the current data to be labeled.
[0130] Since blurry images cannot be accurately labeled, it is necessary to detect blurry images. Optionally, blur detection based on the Laplacian operator can be used to obtain the blur value of each image to be labeled. The blur value of each image to be labeled is then compared with a threshold. If its blur value is greater than the threshold, it means that it is relatively blurry. Among them, dynamic adjustment of the threshold is supported, so that a hierarchical threshold library can be preset for different scenes to meet the needs of different scenes.
[0131] S302 , using a multi-scale sliding window to recheck the local blurriness of each image to be labeled whose blurriness is greater than the blurriness threshold.
[0132] It should be noted that step S301 performs a global rapid detection. To avoid global misjudgments, each image to be annotated with a blur greater than the blur threshold requires a local recheck. Specifically, a multi-scale sliding window is used to select local regions of the image and then perform blur detection on them.
[0133] S303: Remove any image to be labeled whose blurriness of a local area is lower than a blurriness threshold.
[0134] If the blurriness of any local area in the current image to be annotated is lower than the blurriness threshold, it means that there is indeed a large blurriness problem, that is, there is a blur problem, so it needs to be removed.
[0135] S304 : For each remaining image to be annotated, convert the image to be annotated into the HSV color space, extract the data of the brightness channel, and construct a brightness histogram.
[0136] It should be noted that, considering that there may be low-quality images with abnormal exposure, the blurred image is first converted to the HSV color space, and the data of the brightness channel is extracted to construct a 256-level brightness histogram, thereby obtaining the brightness data of the image to be labeled.
[0137] S305 , calculating the pixel ratio of each overexposed area and the pixel ratio of each underexposed area of the image to be labeled according to the brightness histogram.
[0138] Optionally, the overexposed area may refer to an area with a brightness greater than 240, and the underexposed area may refer to an area with a brightness less than 15.
[0139] S306: Mark the image to be annotated that has any overexposed area or underexposed area with a pixel ratio greater than a preset ratio as an exposure abnormality and remove it.
[0140] Optionally, overexposed areas and underexposed areas with a pixel ratio greater than 20% may be marked as abnormal exposure areas, and when any area in the image to be annotated is an abnormal exposure area, it is marked as abnormal exposure.
[0141] S307: Scale and grayscale the remaining images to be labeled.
[0142] In order to facilitate subsequent labeling and calculation of fingerprint hash values to remove duplicate images, the remaining images to be labeled are first scaled and grayscaled. Optionally, the images can be scaled to 32x32 size and grayscaled.
[0143] S308 , extracting low-frequency components of each grayscale image to be labeled by discrete cosine transform, and generating fingerprint hash values of each image to be labeled by using the low-frequency components of each image to be labeled.
[0144] Optionally, a low-frequency component may be extracted through DCT transformation to generate a 64-bit fingerprint hash value.
[0145] S309 : Calculate the distance between each two images to be labeled based on the fingerprint hash values of each two images to be labeled.
[0146] Optionally, the distance between every two images to be annotated may be a Hamming distance, that is, the Hamming distance is used to compare the hash fingerprint.
[0147] S310: Deduplication of images to be labeled whose distance is less than a preset distance.
[0148] Images to be annotated that are closer than a preset distance are considered identical, so duplicates need to be removed. Alternatively, for large-scale images, locality-sensitive hashing (LSH) can be combined to construct a bucket index to reduce the comparison complexity.
[0149] Optionally, in another embodiment of the present application, when the current data to be labeled is video data, a specific implementation of step S105 is as follows: Figure 4 Shown, including:
[0150] S401: Decode each video to be labeled in the current data to be labeled.
[0151] Specifically, for video data, video decoding is first performed to decode the video file into an original frame sequence, thereby facilitating subsequent processing.
[0152] S402: Filter out low-quality videos to be labeled based on the parameters of each video to be labeled.
[0153] Optionally, blurry or damaged videos can be filtered out by checking whether parameters such as video resolution, frame rate, and duration meet requirements.
[0154] S403: Extract key frames from each filtered video to be labeled.
[0155] Optionally, key frames may be automatically extracted based on changes in video content or by setting an interval for key frame extraction, thereby effectively reducing the workload of annotation.
[0156] Optionally, in another embodiment of the present application, when the current data to be labeled is audio data, a specific implementation of step S105 is as follows: Figure 5 Shown, including:
[0157] S501: Convert all audio files to be annotated in the current data to be annotated into a unified format.
[0158] Optionally, audio data in non-PCM format can be converted to WAV (16kHz / 16bit / mono), which can effectively reduce the adaptation complexity of the annotation process.
[0159] S502: performing noise reduction and enhancement on each audio to be annotated after the format conversion.
[0160] Optionally, the proportion of valid labeled data can be increased by integrating FFmpeg noise reduction filters and VAD (voice activity detection) to automatically remove silent segments.
[0161] S503: Based on at least the decodability, effective duration, and sampling rate consistency of each audio to be labeled after noise reduction and enhancement, screen out and filter out audio to be labeled that does not meet the requirements.
[0162] S106 , based on the current scene type and the type of the current data to be labeled, calling the corresponding labeling model to analyze the pre-processed current data to be labeled, and outputting the labeling of the current data to be labeled and its confidence level.
[0163] Among them, the current data to be annotated of the non-text data is analyzed based on its annotated relationship data, that is, it depends on its annotated relationship data.
[0164] Optionally, in an embodiment of the present application, models developed using different machine learning frameworks are integrated and supported through back-end services, such as models built using different frameworks such as PyTorch, TensorFlow, scikit-learn, and Hugging Face. In addition, the most suitable tool can be selected from a variety of machine learning frameworks to build models based on the needs of specific tasks. For example, for natural language processing tasks, Hugging Face provides a wealth of pre-trained models and tools to facilitate tasks such as text classification and named entity recognition. For image recognition tasks, PyTorch or TensorFlow are more advantageous. They have powerful image processing libraries and efficient computing power, which improves the flexibility of model selection. By supporting cross-framework model compatibility, existing model resources can be integrated, avoiding duplicate development work and improving resource reusability. In addition, by selecting different models for different scenarios, the overall stability and reliability of the system can be improved, providing strong automated support for labeling tasks.
[0165] Therefore, you need to first determine the required model based on the current scenario and the type of data to be labeled, and then call it. The data to be labeled is then fed into the called annotation model, which analyzes the data and predicts the label for it. After the prediction is complete, the model's prediction output format is standardized, requiring the return of a confidence score (such as the maximum Softmax value for classification tasks or the mean Box Score value for detection tasks) and setting a confidence threshold.
[0166] S107: Determine whether the confidence level of the annotation of the current data to be annotated is lower than a preset threshold.
[0167] It should be noted that if the confidence level of the annotation of the current data to be annotated is not lower than the preset threshold, it means that the annotation result is accurate and can be directly output. If the confidence level of the annotation of the current data to be annotated is lower than the preset threshold, it means that there is a problem with the annotation result and manual review and correction are required. In this case, step S108 is executed.
[0168] S108: triggering a review process for the annotation of the current data to be annotated.
[0169] Optionally, the data can be stored in an intermediate table and subsequently sent to relevant personnel for manual review through the corresponding review process.
[0170] S109: When a target event is monitored, a fine-tuning interface call is triggered to fine-tune the annotation model using the reviewed current data to be annotated and its annotations through the fine-tuning interface.
[0171] The target event refers to the event that updates the annotation of the data.
[0172] It should be noted that if the data is modified after review, it means that the labeling results output by the model are wrong. In order to improve the accuracy of the labeling model, the current data to be labeled and its labeling after review should be used to fine-tune the labeling model in a timely manner.
[0173] Alternatively, a webhook event-driven model can be employed to automatically trigger fine-tuning API calls based on annotation lifecycle events (such as annotation creation or annotation update). This can push newly annotated data to the training cluster to initiate model iterations and trigger quality checks when anomalies in annotation results occur. Furthermore, because training is incremental, new annotated data can be used to further train an existing model without having to train the entire model from scratch, saving time and computing resources. As the amount of annotated data continues to grow, the model can be continuously optimized through continuous training, thereby improving the accuracy and consistency of annotations.
[0174] An embodiment of the present application provides a multimodal data annotation method, which obtains current data to be annotated, which may include data from various modalities. If the current data to be annotated is text data, the current data to be annotated is imported. If the current data to be annotated is non-text data, the annotation relationship data of the current data to be annotated is analyzed based on the type of the current data to be annotated, and the current data to be annotated and its annotation relationship data are imported so that accurate annotation of the corresponding data can be achieved based on the annotation relationship data. Then, based on the type of the current data to be annotated, the current data to be annotated is preprocessed using a corresponding preprocessing method to filter out problematic data, etc., and data that can be subsequently analyzed can be obtained. Then, based on the current scene type and the type of the current data to be annotated, a corresponding annotation model is called to analyze the preprocessed current data to be annotated, and the annotation and confidence level of the current data to be annotated are output. The current data to be annotated for non-text data is analyzed based on its annotation relationship data. Thus, based on the scene type and the type of data, the corresponding annotation model is called to perform annotation prediction, thereby adapting to various scene types and data types and ensuring the accuracy of annotation. Furthermore, if the confidence level of the annotation of the current data to be annotated is lower than a preset threshold, the audit process for the annotation of the current data to be annotated is triggered, so that a small amount of erroneous data that may exist in the model annotation can be audited and promptly corrected. Finally, when the target event is monitored, the fine-tuning interface call is triggered to fine-tune the annotation model using the audited current data to be annotated and its annotations through the fine-tuning interface. Among them, the target event refers to the event that updates the annotation of the data, so that the adjusted data can be used to fine-tune the annotation model in a timely manner, thereby further ensuring the accuracy of the annotation model's annotation of the data, thereby realizing a method for efficiently and accurately annotating data.
[0175] Another embodiment of the present application provides a multimodal data annotation system, such as Figure 6 Shown, including:
[0176] The data acquisition unit 601 is used to acquire the current data to be labeled.
[0177] The first importing unit 602 is configured to import the current data to be annotated when the current data to be annotated is text data.
[0178] The second importing unit 603 is used to analyze the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated when the current data to be annotated is non-text data, and import the current data to be annotated and its annotation relationship data.
[0179] The preprocessing unit 604 is configured to preprocess the data to be labeled using a corresponding preprocessing method according to the type of the data to be labeled.
[0180] The annotation unit 605 is configured to analyze the pre-processed data to be annotated by calling a corresponding annotation model based on the current scene type and the type of the data to be annotated, and output the annotation and confidence level of the data to be annotated. For non-text data, the data to be annotated is analyzed based on its annotation relationship data.
[0181] The review unit 606 is configured to trigger a review process for the labeling of the current data to be labeled when the confidence level of the labeling of the current data to be labeled is lower than a preset threshold.
[0182] The fine-tuning unit 607 is configured to trigger a fine-tuning interface call when a target event is detected, so as to fine-tune the annotation model using the reviewed current data to be annotated and its annotations through the fine-tuning interface.
[0183] Optionally, in a multimodal data annotation system according to another embodiment of the present application, when the second importing unit analyzes the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated, it is configured to:
[0184] If the current data to be annotated is image data, an adapter is used to identify a annotation file that matches the current data to be annotated.
[0185] If the current data to be labeled is video data, a labeling file matching the current data to be labeled is identified.
[0186] Convert the format of the current data to be annotated and parse the matching annotation file.
[0187] The current data to be annotated after format conversion is mapped to the annotated file data obtained by parsing on a time axis to obtain the annotated file data that matches each frame image in the current data to be annotated.
[0188] If the current data to be labeled is video data, basic information extraction, multi-audio track processing and ancillary data analysis operations are performed on the current data to be labeled.
[0189] Optionally, in a multimodal data annotation system in another embodiment of the present application, when the current data to be annotated is text data, the preprocessing unit performs preprocessing on the current data to be annotated using a corresponding preprocessing method according to the type of the current data to be annotated, and is used to:
[0190] Perform unified encoding conversion on the current data to be labeled.
[0191] The structure of the current data to be labeled after encoding conversion is standardized.
[0192] Add corresponding language tags to the current data to be annotated after structural standardization.
[0193] Perform data cleaning and quality verification on the current data to be annotated after adding language tags.
[0194] Optionally, in a multimodal data annotation system of another embodiment of the present application, when the current data to be annotated is image data, the preprocessing unit performs preprocessing on the current data to be annotated using a corresponding preprocessing method according to the type of the current data to be annotated, and is used to:
[0195] Perform blur detection on each image to be labeled in the current data to be labeled.
[0196] A multi-scale sliding window is used to recheck the local ambiguity of each image to be labeled whose ambiguity is greater than the ambiguity threshold.
[0197] Remove any image to be labeled whose blurriness of a local area is lower than the blurriness threshold.
[0198] For each remaining image to be labeled, convert the image to the HSV color space, extract the data of the brightness channel, and construct a brightness histogram.
[0199] According to the brightness histogram, the pixel ratio of each overexposed area and the pixel ratio of each underexposed area of the image to be labeled are calculated respectively.
[0200] The image to be annotated with any overexposed area or underexposed area whose pixel ratio is greater than a preset ratio is marked as exposure abnormality and removed.
[0201] The remaining images to be labeled are scaled and grayscaled.
[0202] The low-frequency components of each grayscale image to be labeled are extracted by discrete cosine transform, and the fingerprint hash value of each image to be labeled is generated using the low-frequency components of each image to be labeled.
[0203] Based on the fingerprint hash values of each two images to be annotated, the distance between each two images to be annotated is calculated.
[0204] The images to be annotated whose distance is less than the preset distance are deduplicated.
[0205] Optionally, in a multimodal data annotation system according to another embodiment of the present application, when the current data to be annotated is video data, the preprocessing unit performs preprocessing on the current data to be annotated using a corresponding preprocessing method according to the type of the current data to be annotated, and is configured to:
[0206] Decode each video to be labeled in the current data to be labeled.
[0207] Based on the parameters of each video to be labeled, low-quality videos to be labeled are filtered out.
[0208] Extract key frames from each filtered video to be labeled.
[0209] Optionally, in a multimodal data annotation system according to another embodiment of the present application, when the current data to be annotated is audio data, the preprocessing unit performs preprocessing on the current data to be annotated using a corresponding preprocessing method according to the type of the current data to be annotated, and is configured to:
[0210] Convert all the audio files to a unified format in the current data to be annotated.
[0211] Noise reduction and enhancement are performed on each audio to be annotated after the format conversion.
[0212] At least based on the decodability, effective duration, and sampling rate consistency of each audio to be labeled after noise reduction and enhancement, the audio to be labeled that does not meet the requirements is screened out and filtered out.
[0213] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.
[0214] Another embodiment of the present application provides an electronic device, such as Figure 7 Shown, including:
[0215] Memory 701 and processor 702 .
[0216] The memory 701 is used to store programs.
[0217] The processor 702 is used to execute the program stored in the memory 701. When the program is executed, it is specifically used to implement the multimodal data labeling method provided in any one of the above embodiments.
[0218] Another embodiment of the present application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the multimodal data annotation method provided in any of the above embodiments.
[0219] Computer storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.
[0220] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0221] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A multimodal data annotation method, characterized in that: include: Get the current data to be labeled; If the current data to be annotated is text data, importing the current data to be annotated; If the current data to be annotated is non-text data, analyzing the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated, and importing the current data to be annotated and its annotation relationship data; Preprocessing the data to be labeled using a corresponding preprocessing method according to the type of the data to be labeled; According to the current scene type and the type of the current data to be annotated, calling the corresponding annotation model to analyze the preprocessed current data to be annotated, and outputting the annotation of the current data to be annotated and its confidence; wherein the current data to be annotated for non-text data is analyzed based on its annotation relationship data; If the confidence level of the annotation of the current data to be annotated is lower than a preset threshold, a review process is triggered for the annotation of the current data to be annotated; When a target event is monitored, a fine-tuning interface call is triggered to fine-tune the annotation model using the reviewed current data to be annotated and its annotations through the fine-tuning interface; wherein the target event refers to an event that updates the annotation of the data.
2. The method according to claim 1, characterized in that The step of analyzing the labeling relationship data of the current data to be labeled according to the type of the current data to be labeled includes: If the current data to be annotated is image data, identifying a marking file that matches the current data to be annotated through an adapter; If the current data to be labeled is video data, identifying a labeling file that matches the current data to be labeled; Convert the format of the current data to be annotated and parse the matching annotation file; Performing time axis mapping on the current data to be annotated after format conversion and the annotated file data obtained by parsing, to obtain the annotated file data that matches each frame image in the current data to be annotated; If the current data to be labeled is video data, basic information extraction, multi-track processing, and attached data analysis operations are performed on the current data to be labeled.
3. The method according to claim 1, characterized in that The current data to be annotated is text data, and the preprocessing of the current data to be annotated by using a corresponding preprocessing method according to the type of the current data to be annotated includes: Performing unified coding conversion on the current data to be annotated; Performing structural standardization on the current data to be annotated after encoding conversion; Adding corresponding language tags to the current data to be annotated after structural standardization; The data to be annotated after the language tags are added is cleaned and quality checked.
4. The method according to claim 1, wherein The current data to be labeled is image data, and the preprocessing of the current data to be labeled by using a corresponding preprocessing method according to the type of the current data to be labeled includes: Performing blur detection on each image to be labeled in the current data to be labeled; Rechecking the local fuzziness of each of the to-be-annotated images whose fuzziness is greater than the fuzziness threshold using a multi-scale sliding window; Removing the image to be labeled if the blurriness of any local area is lower than the blurriness threshold; For each of the remaining images to be annotated, convert the image to be annotated into the HSV color space, extract the data of the brightness channel, and construct a brightness histogram; Calculating the pixel ratio of each overexposed area and the pixel ratio of each underexposed area of the image to be labeled according to the brightness histogram; Marking the image to be marked as having an exposure abnormality and removing it if any of the overexposed areas or the underexposed areas has a pixel ratio greater than a preset ratio; Resizing and graying the remaining images to be labeled; Extracting low-frequency components of each grayscale image to be labeled by discrete cosine transform, and generating fingerprint hash values of each image to be labeled by using the low-frequency components of each image to be labeled; Calculating the distance between each two images to be annotated based on the fingerprint hash values of each two images to be annotated; Deduplication is performed on the images to be labeled whose distance is less than a preset distance.
5. The method according to claim 1, characterized in that The current data to be labeled is video data, and the preprocessing of the current data to be labeled by using a corresponding preprocessing method according to the type of the current data to be labeled includes: Decoding each video to be labeled in the current data to be labeled; Filtering out low-quality videos to be labeled based on parameters of each video to be labeled; Key frames are extracted from each of the filtered videos to be labeled.
6. The method according to claim 1, characterized in that The current data to be labeled is audio data, and the preprocessing of the current data to be labeled by using a corresponding preprocessing method according to the type of the current data to be labeled includes: Converting the audio files to a unified format in the current data to be annotated; Performing noise reduction and enhancement on each of the audios to be annotated after the format conversion; Based at least on the decodability, effective duration, and sampling rate consistency of each of the audios to be labeled after noise reduction and enhancement, the audios to be labeled that do not meet the requirements are screened out and filtered out.
7. A multimodal data annotation system, characterized in that: include: A data acquisition unit, used to acquire the current data to be labeled; A first importing unit, configured to import the current data to be annotated when the current data to be annotated is text data; A second importing unit is configured to analyze the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated when the current data to be annotated is non-text data, and import the current data to be annotated and its annotation relationship data; A preprocessing unit, configured to preprocess the current data to be labeled using a corresponding preprocessing method according to the type of the current data to be labeled; a labeling unit, configured to analyze the preprocessed data to be labeled by calling a corresponding labeling model according to the current scene type and the type of the data to be labeled, and output the labeling of the data to be labeled and its confidence level; wherein the data to be labeled that is non-text data is analyzed based on its labeling relationship data; An audit unit, configured to trigger an audit process for the annotation of the current data to be annotated when the confidence level of the annotation of the current data to be annotated is lower than a preset threshold; A fine-tuning unit is used to trigger a fine-tuning interface call when a target event is monitored, so as to fine-tune the annotation model using the reviewed current data to be annotated and its annotations through the fine-tuning interface; wherein the target event refers to an event that updates the annotation of the data.
8. The system according to claim 7, characterized in that When the second importing unit performs the analysis of the annotation relationship data of the current data to be annotated according to the type of the current data to be annotated, it is used to: If the current data to be annotated is image data, identifying a marking file that matches the current data to be annotated through an adapter; If the current data to be labeled is video data, identifying a labeling file that matches the current data to be labeled; Convert the format of the current data to be annotated and parse the matching annotation file; Performing time axis mapping on the current data to be annotated after format conversion and the annotated file data obtained by parsing, to obtain the annotated file data that matches each frame image in the current data to be annotated; If the current data to be labeled is video data, basic information extraction, multi-track processing, and attached data analysis operations are performed on the current data to be labeled.
9. An electronic device, characterized in that: include: memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the multimodal data labeling method according to any one of claims 1 to 6.
10. A computer storage medium, characterized in that Used to store a computer program, which, when executed by a processor, is used to implement the multimodal data labeling method according to any one of claims 1 to 6.
Citation Information
Cited By
Multi-modal data labeling method and system based on large model pre-labeling
CN121456489A