Image labeling method, image labeling device and related product

Abstract

The embodiment of the invention discloses an image annotation method, an image annotation device and a related product, and is used for the technical field of image processing. According to the method, a multi-modal large model zero sample is called to detect an instance box of an instance of a first category in a video image, initial image annotation data is obtained, and increase from the zero sample to the initial image annotation data is achieved; when the data size of the initial image annotation data reaches a first number threshold value, based on the initial image annotation data, training to obtain an exclusive model of which the detection precision for a second type of examples is superior to that of a multi-modal large model, and calling the exclusive model to detect a second type of example frames in a large number of preset video images, a large amount of target image annotation data is obtained, and large-scale increase of the image annotation data is realized; automatic labeling from zero sample to high throughput is achieved, manual labeling is not needed in the image labeling process, and the image labeling efficiency can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image annotation method, an image annotation device, and related products. Background Technology

[0002] Existing visual perception models are widely used in fields such as autonomous driving, industrial inspection, and medical imaging. These models can be detection and segmentation models (such as YOLO (You Only Look Once) and DETR (DEtection Transformer)) and multimodal large models (such as Gemma3 and Qwen3-VL). Before using a visual perception model, it needs to be trained using image-annotated data.

[0003] In recent years, with the significant development and rapid market penetration of visual perception models, the amount of image annotation data required for training these models has grown exponentially. Currently, a certain amount of image annotation templates often needs to be manually annotated before rules or algorithms can be used to annotate image data based on these templates, resulting in a large volume of annotated image data. However, because image annotation data requires annotation at the pixel level, the annotation is quite difficult; relying on manual annotation templates during the image annotation process leads to low annotation efficiency and makes it difficult to quickly annotate large-scale image data. Summary of the Invention

[0004] This application provides an image annotation method, an image annotation device, and related products, which can quickly annotate large-scale image data.

[0005] This application provides an image annotation method, including:

[0006] Obtain video images containing instances of a first category from the original video, where the first category contains multiple categories;

[0007] The multimodal large model is invoked to detect the first category of instance boxes in the video image with zero samples, and the instance boxes in the video image are labeled with pixel-level masks to obtain the initial image annotation data;

[0008] When the amount of initial image annotation data reaches a first quantity threshold, a specialized model with better instance detection accuracy for the second category than the multimodal large model is trained based on the initial image annotation data; wherein, the second category is any category in the first category;

[0009] The specialization model is invoked to detect the second category of instance bounding boxes in the preset video image, and the instance bounding boxes of the preset video image are labeled with pixel-level masks to obtain target image annotation data; wherein, the number of image frames of the preset video image is greater than a preset frame number threshold.

[0010] Furthermore, obtaining video images containing instances of the first category from the original video includes:

[0011] The original video is cleaned to obtain a cleaned video image;

[0012] The cleaned video images and instance information of the first category are input into the multimodal retrieval model to obtain video images containing instances of the first category selected by the multimodal retrieval model from the cleaned video images.

[0013] Furthermore, the step of calling the multimodal large model for zero-shot detection of the first category of instance boxes in the video image includes:

[0014] The video image and the category information of the first category are input into the multimodal large model to obtain the initial instance bounding box of the first category in the video image detected by the multimodal large model with zero samples;

[0015] The initial instance boxes are deduplicated to obtain instance boxes of the first category in the video image.

[0016] Furthermore, the video images include: multiple frames of video images in temporal sequence;

[0017] The step of annotating the instance bounding boxes of the video image with pixel-level masks to obtain initial image annotation data includes:

[0018] The instance bounding boxes of the multi-frame video images are input into the image segmentation model, and the image pixels contained in the instance bounding boxes of the multi-frame video images are labeled with temporal masks based on the image segmentation model.

[0019] The initial image annotation data is obtained by removing the temporal masks in the multi-frame video images whose overlap is greater than a preset overlap threshold and whose integrity is less than a preset integrity threshold.

[0020] Furthermore, the step of training a specialized model based on the initial image annotation data, which has better instance detection accuracy for the second category than the multimodal large model, includes:

[0021] In each training cycle, the initial image annotation data is divided into a training set and a validation set;

[0022] After training the initial specialized model on the training set, the first detection accuracy of the initial specialized model for the second category and the second detection accuracy of the multimodal large model for the second category are determined on the validation set.

[0023] The first detection accuracy is compared with the second detection accuracy until the first detection accuracy is greater than the second detection accuracy, thus obtaining the trained specialized model.

[0024] Furthermore, the step of annotating the instance bounding boxes of the preset video image with pixel-level masks to obtain target image annotation data includes:

[0025] The multimodal large model is invoked to detect instance boxes of long-tail categories in the preset video image, wherein the long-tail category is a category other than the second category in the first category;

[0026] Pixel-level masks are used to annotate the instance bounding boxes of the second category and the instance bounding boxes of the long-tail category in the preset video image to obtain target image annotation data.

[0027] Furthermore, after annotating the instance bounding boxes of the preset video image with pixel-level masks to obtain the target image annotation data, the method further includes:

[0028] When the amount of data of the target image annotation data is greater than the second quantity threshold, any frame of the video image in the preset video image is input into the expert model group to obtain the detection results of the multiple models in the expert model group for the any frame of the video image; the model in the expert model group is any one or two of the multimodal large model and the specialized model.

[0029] The detection difficulty of any frame of video image is determined based on the error between the detection results of any two models in the expert model group for any frame of video image.

[0030] From the preset video images, identify difficult video images whose detection difficulty is greater than a preset difficulty threshold;

[0031] The specialized model is incrementally trained based on the image annotation data of the difficult example video images, and the incrementally trained specialized model is then used for image annotation.

[0032] Furthermore, the detection result of the model in the expert model group for any frame of video image includes: the pixel-level mask corresponding to the instance box detected by the model in the expert model group for any frame of video image;

[0033] The determination of the detection difficulty of any given video frame based on the error between the detection results of any two models in the expert model group for any given video frame includes:

[0034] Use the pixel-level mask of one of the two models corresponding to any frame of video image as the prediction mask, and use the pixel-level mask of the other model corresponding to any frame of video image as the ground mask.

[0035] The intersection-over-union ratio (IoU) of any frame of video image is calculated based on the predicted mask and the real mask.

[0036] The detection difficulty of any given video frame is determined based on the cross-union ratio (CUP) of that frame.

[0037] This application also provides an image annotation apparatus, including:

[0038] The acquisition unit is used to acquire video images containing a first category in the original video, wherein the first category contains multiple categories;

[0039] The first annotation unit is used to call the multimodal large model to detect the instance boxes of the first category in the video image with zero samples, and to annotate the instance boxes of the video image with pixel-level masks to obtain the initial image annotation data;

[0040] The training unit is used to train a specialized model with better instance detection accuracy for the second category than the multimodal large model based on the initial image annotation data when the amount of the initial image annotation data reaches a first quantity threshold; wherein, the second category is any category in the first category;

[0041] The second annotation unit is used to call the specialized model to detect the instance boxes of the second category in the preset video image, and to annotate the instance boxes of the preset video image with pixel-level masks to obtain target image annotation data; wherein, the number of image frames of the preset video image is greater than a preset frame number threshold.

[0042] This application also provides an electronic device, including:

[0043] Processor, memory, input / output interfaces, wired or wireless network interfaces, power supply;

[0044] The memory is either a short-term storage memory or a persistent storage memory;

[0045] The processor is configured to communicate with the memory and execute instructions in the memory on a control plane functional entity to perform the methods described above.

[0046] This application also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to perform the method described above.

[0047] This application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described above.

[0048] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0049] In this embodiment, video images containing instances of a first category are obtained from the original video, where the first category includes multiple categories; a multimodal large model is invoked to detect instance boxes of the first category in the video images with zero-shot detection, and pixel-level masks are added to the instance boxes of the video images to obtain initial image annotation data; when the amount of initial image annotation data reaches a first quantity threshold, a specialized model with better instance detection accuracy for the second category than the multimodal large model is trained based on the initial image annotation data; wherein, the second category is any category in the first category; the specialized model is invoked to detect instance boxes of the second category in a preset video image, and pixel-level masks are added to the instance boxes of the preset video image to obtain target image annotation data; wherein, the number of frames in the preset video image is greater than a preset frame number threshold.

[0050] As can be seen, in this embodiment, the multimodal large model is used to detect the first category of instance boxes in video images with zero samples to obtain initial image annotation data, thus achieving growth from zero samples to initial image annotation data. When the amount of initial image annotation data reaches a first quantity threshold, a specialized model with better accuracy in detecting second category instances than the multimodal large model is trained based on the initial image annotation data. The specialized model is then used to detect a large number of second category instance boxes in a large number of preset video images, resulting in a large amount of target image annotation data, thus achieving a significant increase in image annotation data. In this embodiment, based on the dynamic growth of the amount of image annotation data, the corresponding annotation strategy is automatically switched, realizing automatic annotation from zero samples to high throughput. No manual annotation is required during the image annotation process, which can effectively improve the efficiency of image annotation. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0052] Figure 1 This is a schematic diagram of a communication architecture disclosed in an embodiment of this application;

[0053] Figure 2 This is a flowchart of an image annotation method disclosed in an embodiment of this application;

[0054] Figure 3 This is a flowchart of another image annotation method disclosed in an embodiment of this application;

[0055] Figure 4 This is a flowchart of an incremental training method disclosed in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of a data acquisition stage disclosed in an embodiment of this application;

[0057] Figure 6 This is a schematic diagram of an initial annotation stage disclosed in an embodiment of this application;

[0058] Figure 7 This is a schematic diagram of a data accumulation stage disclosed in an embodiment of this application;

[0059] Figure 8 This is a schematic diagram of a difficult example screening and incremental training stage disclosed in an embodiment of this application;

[0060] Figure 9 This is a schematic diagram of an image annotation device disclosed in an embodiment of this application;

[0061] Figure 10 This is a schematic diagram of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0062] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0063] In the description of the embodiments of this application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0064] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0065] With the significant development and rapid market penetration of visual perception models, the amount of image annotation data required for training these models is growing exponentially. To better understand the image annotation method of this application, the communication scenario of this application is first described, such as... Figure 1 As shown: The image annotation device 101 is communicatively connected to one or more image collection devices 102, and is used to annotate the image data collected by the image collection device 102, and provide the obtained image annotation data to the training process of the visual perception model.

[0066] In the image annotation device 101, a certain number of image annotation templates often need to be manually annotated before rules or algorithms can be used to annotate the image data based on these templates, resulting in a large amount of image annotation data. However, since image annotation data requires annotation at the pixel level, the annotation is quite difficult; relying on manual annotation of image templates during the image annotation process leads to low annotation efficiency and makes it difficult to quickly annotate large-scale image data. Therefore, this application provides an image annotation method that can quickly annotate large-scale image data, such as... Figure 2 As shown, the specific steps include the following:

[0067] 201. Obtain video images containing instances of the first category from the original video.

[0068] In this embodiment, the image annotation device can acquire video images containing instances of the first category from the original video. The original video can be a video captured by a camera or a video from a pre-stored video dataset; specific details are not limited here. This application primarily annotates instances of the first category in the images, requiring the identification and filtering of video images containing instances of the first category from the original video. These video images can be multi-frame video images. The first category includes multiple subcategories. A category refers to an abstract classification of things with similar attributes (shape, function, semantics, etc.), representing the common characteristics of a class of things. The specific category information of the first category can be determined based on the corresponding visual perception scene. For example, if the visual perception scene is autonomous driving, the first category is the object category to be identified in autonomous driving, and the first category can include categories such as pedestrians, vehicles, trees, and signs; specific details are not limited here. If the visual perception scene is industrial inspection, the first category is the category of faulty parts to be identified in industrial inspection, and the first category can include categories such as faulty part A and faulty part B; specific details are not limited here.

[0069] 202. Use the multimodal large model to detect the first category of instance boxes in the video image with zero samples, and annotate the instance boxes of the video image with pixel-level masks to obtain the initial image annotation data.

[0070] After acquiring video images containing the first category from the original video, a multimodal large model can be used for zero-shot detection of the first category's instance boxes in the video images. Pixel-level masks are then applied to these instance boxes to obtain initial image annotation data. This process achieves growth from zero-shot to initial image annotation data without the need for manual image annotation templates. Zero-shot detection refers to detection without using image annotation samples. Specifically, during the cold-start annotation phase, when there is no image annotation data or the amount of image annotation data is insufficient for training a lightweight specialized model, a multimodal large model can be used for zero-shot detection of the first category's instance boxes in the video images. This involves detecting the detection boxes of instances of the first category in the video images. These instances of the first category are specific individuals within a certain category, such as "Zhang San" and "Li Si" appearing in the video image, which are two instances of the "pedestrian" category. The corresponding instance box shape can be rectangular, square, or circular; the specific shape is not limited here. The multimodal large model can be either the GroundingDINO model or the Florence 2 model; the specific shape is not limited here.

[0071] To obtain initial image annotation data, pixel-level masks are used to annotate the bounding boxes of a video image. This can be achieved by inputting the first-category bounding boxes from the video image into a SAM (Segment Anything Model). The SAM model segments the bounding boxes into multiple image pixels, and a mask is used to annotate each pixel within the bounding box. The mask can represent the category of an image pixel using different mask values; for example, "1" represents the first category of the bounding box, and "0" represents other categories. The corresponding image annotation data can be pixel-level image segmentation data containing metadata such as the first category, the mask, and the bounding box boundaries.

[0072] 203. When the amount of initial image annotation data reaches the first quantity threshold, a specialized model with better accuracy in detecting second-class instances than the multimodal large model is trained based on the initial image annotation data.

[0073] As the amount of initial image annotation data increases, when it reaches a first threshold, a specialized model is trained based on this initial data to achieve higher accuracy in detecting second-class instances than the multimodal large model. This first threshold is the amount of data sufficient to train the specialized model; for example, it could be 100 or 200 instances, with no specific limitation here. This specialized model is a lightweight detection model, which can be a YOLO (You Only Look Once) model or a DETR (Detection Transformer) model, with no specific limitation here. This specialized model is designed for detecting instances of the second class, which can be any of the first classes. In autonomous driving, the second class can be pedestrians or vehicles, with no specific limitation here.

[0074] Understandably, when training a specialized model based on initial image annotation data, the model can be trained periodically until its accuracy in detecting second-class instances surpasses that of the multimodal large model. Specifically, as the initial image annotation data grows, whenever the amount of data added reaches a first threshold, second-class image annotation data is extracted from the initial data to train the specialized model, and it is then determined whether the specialized model's accuracy in detecting second-class instances is superior to that of the multimodal large model.

[0075] 204. Call the specialized model to detect the second category of instance boxes in the preset video image, and annotate the instance boxes of the preset video image with pixel-level masks to obtain the target image annotation data.

[0076] After obtaining a specialized model that outperforms the multimodal large model in detecting instances of the second category, the specialized model can be called to detect instance boxes of the second category in preset video images. Pixel-level masks are then used to annotate the instance boxes in the preset video images, yielding the target image annotation data. The preset video images can be other video images besides the original video, and can also be captured by a camera or obtained from a video dataset; specific requirements are not specified here. The frame rate of the preset video images must be greater than a preset frame rate threshold, which can be 50,000 or 60,000 frames; specific requirements are not specified here. In other words, the specialized model is called to detect instances in a large number of preset video images, and pixel-level masks are used to annotate the instance boxes in the preset video images, yielding a large amount of target image annotation data. Specifically, the SAM model can be used to segment the instance boxes in the preset video images and annotate them with pixel-level masks to obtain the target image annotation data.

[0077] Understandably, as the amount of initial image annotation data increases, the specialized model trained on the second category of image annotation data achieves higher accuracy in detecting second category instances than the multimodal large model. At this point, the specialized detection model can replace the multimodal large model for second category instance detection, effectively avoiding missed detections of second category instances.

[0078] As can be seen, in this embodiment, the multimodal large model is used to detect the first category of instance boxes in video images with zero samples to obtain initial image annotation data, achieving growth from zero samples to initial image annotation data. When the amount of initial image annotation data reaches a first quantity threshold, a specialized model with higher accuracy in detecting the second category of instances than the multimodal large model is trained based on the initial image annotation data. The specialized model is then used to detect the second category of instance boxes in a large number of preset video images, obtaining a large amount of target image annotation data, achieving a significant increase in image annotation data. In this embodiment, based on the dynamic growth of the amount of image annotation data, the corresponding annotation strategy is automatically switched, achieving automatic annotation from zero samples to high throughput. No manual annotation is required during the image annotation process, which can effectively improve image annotation efficiency. High efficiency and low cost are achieved throughout the entire data annotation lifecycle.

[0079] In one executable approach, the image annotation method can be widely adapted to various two-dimensional image segmentation tasks and is applicable to scenarios such as autonomous driving, intelligent security, industrial quality inspection, and medical imaging. The models involved (multimodal large models and specialized models) are all open source models, and the implementation does not rely on private black-box models or commercial APIs, which facilitates engineering integration and deployment and has both performance and controllability.

[0080] Furthermore, the image annotation method in the embodiments of this application will be described in detail below, such as... Figure 3 As shown, the specific steps include the following:

[0081] 301. Input the cleaned video images and instance information of the first category into the multimodal retrieval model to obtain video images containing the first category.

[0082] In this embodiment, the original video can be cleaned to obtain cleaned video images. This data cleaning includes, but is not limited to, image deduplication, noise reduction, and contrast enhancement. Specifically, image deduplication can involve extracting feature vectors from each frame of the original video using a BGE (BAAI General Embedding) feature extraction model; constructing a vector database using the Milvus framework; calculating the cosine similarity between the feature vectors of any two frames in the vector database to determine the correlation between any two frames; and then using linear sampling, exponential sampling, or Gaussian sampling based on the correlation to filter out the cleaned video images.

[0083] Next, the cleaned video images and instance information of the first category can be input into the multimodal retrieval model to obtain the video images containing the first category selected by the multimodal retrieval model from the cleaned video images. The instance information of the first category can be instance images or instance text describing the first category; the specifics are not limited here. This multimodal retrieval model can be a Vis-BGE (Visual Bidirectional Generative Encoder) model, which performs multimodal retrieval and matching on the cleaned video images through a multimodal detection model to select the video images containing the first category.

[0084] This step 301 can be performed by the raw data filtering and cleaning module in the image annotation device, and the corresponding data collection stage is as follows: Figure 5 As shown, the original video can be deduplicated to obtain a low frame rate video (i.e., cleaned video image). The Vis-BGE multimodal retrieval model retrieves the video image to be labeled (i.e., video image containing the first category) based on the cleaned video image and the text / image retrieval of the target instance describing the first category.

[0085] As can be seen, by introducing a multimodal retrieval model in this step, the retrieval from text / images of the first category to images is realized, and the video images containing the first category in the original video are filtered out in high throughput, which greatly saves the high time cost of manually reviewing a large number of original data frames.

[0086] 302. Input the video image and the category information of the first category into the multimodal large model to obtain the instance bounding box of the first category in the video image.

[0087] After obtaining the video image containing the first category, the video image and the category information of the first category can be input into the multimodal large model to obtain the initial instance boxes of the first category in the video image detected by the multimodal large model with zero samples. The initial instance boxes are deduplicated, such as removing instance boxes with an intersection over union ratio (IoU) > 0.9 to obtain the instance boxes of the first category in the video image.

[0088] This multimodal large model can be either the GroundingDINO model or the Florence 2 model, and zero-shot detection is performed using the multimodal large model. It's understandable that, compared to the GroundingDINO model, the category information of the first category needs to be input into the model one by one; alternatively, the Florence 2 model can be used, which does not require inputting the category information of the first category into the model. In this case, the Florence 2 model can detect instance boxes of all categories in the video image, and the initial instance box of the first category can be selected from all the instance boxes of all categories.

[0089] As can be seen, this step proposes to use a multimodal large model to detect target instances, which can achieve a target instance detection rate of over 60% under zero-sample conditions, and solves the problem of relying on a large amount of manual annotation in the cold start data annotation stage of existing technologies.

[0090] 303. Based on the image segmentation model, the image pixels contained in the instance boxes in multi-frame video images are labeled with temporal masks to obtain the initial image labeling data.

[0091] It is understandable that the original video containing the first category of video images includes: multiple temporally consecutive video images; initial image annotation data can be obtained by annotating the image pixels contained in the instance boxes of the multiple video images with temporal masks based on an image segmentation model, which can be the second version of the SAM model, SAM2. Specifically, the instance boxes of the multiple video images can be input into the image segmentation model, and the image pixels contained in the instance boxes of the multiple video images can be annotated with temporal masks based on the image segmentation model; specifically, an annotation platform with basic data annotation tools (such as CVAT (Computer Vision Annotation Tool)) can be configured; the SAM2 model can be encapsulated and deployed on a server; a serverless service can be deployed through the Nuclio framework, exposing a specific port to the annotation platform to realize the interaction between the annotation platform and the SAM2 model. The SAM2 model segments the image pixels corresponding to the instance boxes in the multiple video images, calls the annotation platform to annotate the image pixels with temporal masks, that is, generates a pixel-level temporal tracking mask for each instance box.

[0092] Understandably, because multimodal large models often detect the same bounding box multiple times in different frames of video images, the mask output by the SAM2 model may have excessive overlap or be incomplete. Therefore, after annotating the image pixels contained in the bounding boxes in multiple frames of video images with temporal masks based on the image segmentation model, temporal masks with overlap greater than a preset overlap threshold and integrity less than a preset integrity threshold in the multiple frames of video images can be removed to obtain initial image annotation data, thus ensuring the quality of the mask and obtaining high-quality initial image annotation data. The overlap greater than the preset overlap threshold can be defined as an Intersection over Union (IoU) ratio > 0.9 or > 0.95, which is not specifically limited here. The integrity less than the preset integrity threshold can be defined as the number of connected components in the mask > 5 or > 6, which is not specifically limited here; that is, the mask contains too many discontinuous regions, resulting in an incomplete mask.

[0093] It is understandable that steps 302 and 303 can be executed using the multimodal large model detection + SAM2 segmentation module in the image annotation device, and the corresponding initial annotation stage is as follows: Figure 6 As shown, the video image to be labeled can be input into a multimodal large model (GroundingDINO model), and instance boxes in the video image can be detected frame by frame. The instance boxes are then input into the SAM2 model to generate temporal tracking masks for each instance box. Deduplication and cleaning are then performed to obtain pre-labeled masks for instance boxes in each frame of the video image, resulting in initial image annotation data. In this embodiment, after obtaining the initial image annotation data, the annotation can be supplemented manually. The image annotation method is a data-driven semi-automatic annotation method for segmentation of two-dimensional (2D) video images. Annotators can use interactive tools like SAM (supporting point selection, box selection, and pen stroke operations) and annotation tools like polygon lasso to supplement any missed targets. If the same instance box appears consecutively in adjacent frames of the video image, SAM2 can be called for temporal annotation to reduce repetitive operations. Furthermore, masks with insufficiently precise boundaries can be refined.

[0094] It is evident that, in the initial annotation stage, by introducing a multimodal large model and the SAM2 model, only a small amount of manual operation is required to obtain high-quality annotation data of over a thousand pixels in a short time; the manual correction rate is significantly lower than that of traditional manual full-process annotation schemes, and the human-machine collaboration efficiency in the initial annotation stage is superior to existing technologies.

[0095] 304. Train and verify the detection accuracy of the initial specialized model based on the initial image annotation data until the detection accuracy of the initial specialized model is greater than that of the multimodal large model, and obtain the trained specialized model.

[0096] As the initial image annotation data increases in scale, the detection accuracy of the initial specialized model can be trained and validated based on the initial image annotation data until the detection accuracy of the initial specialized model is greater than that of the multimodal large model, thus obtaining a fully trained specialized model. That is, a specialized model for the second category can be periodically trained until the specialized model's instance detection accuracy for the second category is superior to that of the multimodal large model for the second category, thus obtaining a fully trained specialized model.

[0097] Specifically, in each training cycle, the initial image annotation data is divided into a training set and a validation set. This can be done by dividing the initial image annotation data equally between the training and validation sets or by other proportions; the specific division is not limited here. Next, an initial specialization model is trained on the training set. This initial specialization model can be a YOLOv8 model. During the training process, the training set can be input into the initial specialization model to obtain the instance bounding box information detected by the initial specialization model. A loss function is constructed based on the instance bounding box information and the training set until the loss function converges, resulting in the trained initial specialization model. After training the initial specialization model on the training set, the first detection accuracy of the initial specialization model for the second category and the second detection accuracy of the multimodal large model for the second category are determined on the validation set. The validation set can be input into both the initial specialized model and the multimodal large model. The detection accuracy is calculated using the masks corresponding to the bounding boxes detected by the initial specialized model and the multimodal large model, respectively. This detection accuracy can be the mean Intersection over Union (mIoU) or the average accuracy (mAP50) at an IoU threshold of 0.5; the specific method is not limited here. The first detection accuracy is compared with the second detection accuracy until the first detection accuracy is greater than the second detection accuracy, resulting in the trained specialized model. The specialized model is then precisely trained based on quantifiable detection accuracy metrics, and the annotation strategy is automatically adjusted based on the trained specialized model.

[0098] As can be seen, in this step, after the initial image annotation data scale is expanded, a dedicated model detection scheme can be switched to make up for the problem that the multimodal large model cannot meet the annotation requirements of the second category, significantly increasing the number of annotation masks per unit time and greatly reducing the annotation cost.

[0099] 305. Use the dedicated model to detect the second category of instance boxes in the preset video image, and use the multimodal large model to detect the long-tail category of instance boxes in the preset video image.

[0100] After obtaining the trained specialized model, it can be used to replace the multimodal large model to detect the second category of instance boxes in the preset video images, and the multimodal large model can be used to detect the long-tail category of instance boxes in the preset video images. The long-tail category refers to the category other than the second category in the first category.

[0101] Understandably, the long-tail category refers to the category that appears less frequently in the original video. Due to the difficulty in controlling the uniform distribution of each category during data acquisition, there are usually a few categories with a limited number of instances, which are the long-tail categories. In video images, the number of long-tail category instances is far less than that of the second category, resulting in a lack of corresponding labeled samples for the long-tail category when training the specialized model. This leads to poor detection accuracy of the trained specialized model for long-tail category instances. In this case, it is necessary to call a multimodal large model to supplement the detection of instance boxes of the long-tail category in the preset video images, thereby reducing the time cost of manually supplementing the instances missed by the specialized model.

[0102] 306. Label the instance bounding boxes of the second category and the instance bounding boxes of the long-tail category in the preset video image with pixel-level masks to obtain the target image annotation data.

[0103] Next, pixel-level masks can be applied to the bounding boxes of the second category and the long-tail category in the preset video image to obtain the target image annotation data. Specifically, the bounding boxes of the second category and the long-tail category in the preset video image can be input into the SAM model to generate a corresponding tracking mask for each bounding box in the temporal dimension. Furthermore, masks with excessive overlap or incompleteness can be removed to ensure mask quality.

[0104] It is understandable that steps 304 to 306 can be executed by the specialized model and multimodal large model collaborative detection + SAM2 segmentation module in the image annotation device, and the corresponding data accumulation stage is as follows: Figure 7 As shown, the distribution of continuous video frames in a preset video image can be input into a dedicated model (YOLOv8 model) and a multimodal large model (GroundingDINO model) to obtain the second category detection box detected by the dedicated model and the long-tail category detection box detected by the multimodal large model. The second category detection box and the long-tail category detection box are then input into the SAM2 model to generate a temporal tracking mask. After deduplication and cleaning, a pre-annotated mask is obtained, resulting in the target image annotation data.

[0105] As can be seen, during the data accumulation stage, considering that the multimodal large model cannot meet the customized annotation requirements of various categories, a method of collaboration between the dedicated detection model and the multimodal large model is proposed. This method utilizes the data understanding ability of the dedicated detection model and the excellent generalization ability of the multimodal large model for long-tail categories, thereby further improving the automation level of video image segmentation and annotation.

[0106] Furthermore, when the detection accuracy of the specialized model approaches saturation across all categories, the annotation strategy can shift from frame-by-frame annotation to hard example selection. Samples with significant prediction errors are selected as hard examples to accumulate data on long-tail categories. Incremental training further improves the generalization ability of the specialized model. The corresponding hard example selection and incremental training stages are as follows: Figure 4 As shown in the figure, the steps can be performed by the hard example and incremental training module in the image annotation device. The specific steps are as follows:

[0107] 401. When the amount of data of the target image annotation data is greater than the second quantity threshold, input any frame of video image in the preset video image into the expert model group to obtain the detection results of multiple models in the expert model group for any frame of video image.

[0108] When the amount of target image annotation data exceeds a second threshold, any frame of video image from a preset video image is input into the expert model group to obtain the detection results of multiple models in the expert model group for that frame of video image. The amount of target image annotation data exceeding the second threshold can be defined as an increase of 10,000 or 20,000 annotation data points in the target image annotation data; the specific threshold is not limited here. At this point, the detection accuracy gain of the specialized model tends to stabilize or decrease with the increase in data, requiring incremental training of the specialized model. The models in this expert model group can be any one or both of the following: a multimodal large model and a specialized model; for example, the models in this expert model group can be multiple multimodal large models, multiple specialized models, or a combination of multimodal large models and specialized models.

[0109] Any frame of video image can be input into multiple models in the expert model group to obtain the instance bounding boxes in the video image detected by the multiple models. The mask of the instance bounding boxes in the video image is then obtained, which is the detection result of the video image.

[0110] 402. Based on the error between the detection results of any two models in the expert model group for any frame of video image, determine the detection difficulty of any frame of video image.

[0111] Next, the detection difficulty of any given video frame can be determined based on the error between the detection results of any two models in the expert model group for any given video frame. This error can be quantified as the mean Intersection over Union (mIoU) or mAP50, without specific limitations here. By quantifying the error between detection results and determining the detection difficulty of any given video frame, difficult video images containing edge scenes, severe occlusion, or long-tail categories can be accurately selected. Including these selected difficult video images in the incremental training process can guide the specialized model to quickly adapt and robustly enhance the weak regions corresponding to these difficult video images, thereby improving the performance of the specialized model in long-tail categories and complex scenes, and enhancing the generalization ability of the specialized model.

[0112] Specifically, the pixel-level mask of one of the two models corresponding to any frame of video image can be used as the prediction mask, and the pixel-level mask of the other model corresponding to any frame of video image can be used as the ground truth mask; the intersection-over-union (IoU) ratio of any frame of video image can be calculated based on the prediction mask and the ground truth mask; the detection difficulty of any frame of video image can be determined based on the IoU ratio of any frame of video image, that is, the smaller the IoU ratio, the more difficult it is to detect any frame of video image.

[0113] The intersection-over-union ratio of the corresponding i-th frame of the video image ,

[0114] in, Let be the predicted mask for the i-th frame of the video image. Let be the true mask of the i-th frame of the video image. The intersection-over-union ratio (IoU) represents the similarity between the detection results of any two models. The detection difficulty of any given frame of the video image can be determined by calculating the average IoU of each pair of models in the expert model group for that given frame of the video image; that is, for the detection results of all models in the expert model group for that given frame of the video image, calculate the IoU of each pair to obtain multiple IoU, and take the average of the multiple IoU to obtain the detection difficulty of that given frame of the video image.

[0115] The average cross-union ratio of the i-th frame of the video image can be: , where N is the number of crossover ratios.

[0116] 403. Identify difficult video images from the preset video images whose detection difficulty is greater than the preset difficulty threshold.

[0117] By determining the detection difficulty of any frame in the preset video images, the detection difficulty of each frame in the preset video images can be obtained. Then, difficult video images with a detection difficulty greater than a preset difficulty threshold can be identified from the preset video images. For example, video images with an average intersection-union ratio greater than a preset threshold are considered difficult video images; where the preset threshold can be 0.65 or 0.6, and the specific value is not limited here.

[0118] 404. Based on the image annotation data of difficult example video images, perform incremental training on the specialized model, and call the incrementally trained specialized model to perform image annotation.

[0119] After identifying difficult example video images, a specialized model can be incrementally trained based on the image annotation data of these images, and then the incrementally trained specialized model can be used for image annotation. In other words, the image annotation data of the difficult example video images can be used as a training set for incremental training of the specialized model. The incrementally trained specialized model can effectively improve the detection accuracy of difficult example video images, meaning it can accurately detect long-tail category bounding boxes from these images.

[0120] Difficult example selection and incremental training phase, such as Figure 8 As shown, any frame of video image from a preset video image can be input into an expert model group, which includes multiple models (model a, model b, and model c). The detection results of each model for any frame of video image can be obtained: segmentation a, segmentation b, and segmentation c. The average similarity of the pairwise detection results (i.e., average crossover ratio) can be calculated to obtain difficult video images with low average similarity. Then, the difficult video images can be added to the training set to incrementally train the specialized model.

[0121] It is understood that in this embodiment, the specialized model can be iteratively trained incrementally by repeating steps 401 to 404 until the detection accuracy of the incrementally trained specialized model meets a preset threshold, and then the specialized model is called for image annotation. The detection accuracy can be calculated by taking the mask corresponding to the instance bounding box detected by the incrementally trained specialized model on the validation set and the hard example set, and then calculating the corresponding mAP50 or mIoU. Meeting the preset threshold for detection accuracy can be, for example, mAP50 ≥ 0.75 or mIoU ≥ 0.65, and is not specifically limited here.

[0122] As can be seen, in this embodiment of the application, by constructing a difficult example screening mechanism based on expert model groups and combining it with incremental training of the dedicated model, the dedicated model can be continuously optimized from zero sample annotation to closed-loop incremental training. With the accumulation of difficult examples, it is expected to maintain high detection accuracy under complex conditions (such as extreme lighting, camera distortion, or long-tail categories), and continuously improve the annotation quality of the dedicated model.

[0123] In summary, the image annotation method of this application adopts a data scale-driven adaptive annotation strategy mechanism, which can dynamically switch the optimal annotation strategy according to the changes in the scale of the annotated data: in the zero-sample stage, a multimodal large model combined with the SAM2 model is used to annotate pixel-level temporal masks to quickly establish initial image annotation data (i.e., the basic annotation set); when the data scale reaches a certain level (the amount of initial image annotation data reaches the first quantity threshold), a dedicated model is automatically introduced for a large number of annotations based on the quantifiable instance detection accuracy to reduce missed detections; after the data gradually saturates (the amount of target image annotation data is greater than the second quantity threshold), the dedicated model is continuously optimized through hard example mining and incremental training, forming a closed-loop improvement mechanism covering the entire life cycle of the data; it realizes the entire process from the cold start annotation stage to the lightweight dedicated model replacement, to the hard example mining and incremental training, ensuring that the optimal annotation strategy is adopted under different data scales, greatly improving annotation efficiency and ensuring annotation quality, and enabling fast and high-quality annotation of large-scale image data.

[0124] The image annotation method of this invention introduces deep learning models such as multimodal retrieval models, large multimodal models, and lightweight specialized models. It proposes a dynamic switching mechanism for detection models and an incremental training method, embedding image annotation strategies adapted to different data volumes into the annotation lifecycle closed loop. Compared with existing technologies, this invention has significant advantages such as high automation, high data annotation efficiency, less redundant data, low labor costs, and steadily improved annotation quality.

[0125] This application also provides an image annotation device, such as... Figure 9 As shown, it includes:

[0126] The acquisition unit 901 is used to acquire video images containing instances of a first category in the original video, wherein the first category contains multiple categories;

[0127] The first annotation unit 902 is used to call the multimodal large model to detect the instance boxes of the first category in the video image with zero samples, and to annotate the instance boxes of the video image with pixel-level masks to obtain the initial image annotation data;

[0128] Training unit 903 is used to train a specialized model with better instance detection accuracy for the second category than the multimodal large model based on the initial image annotation data when the amount of the initial image annotation data reaches a first quantity threshold; wherein, the second category is any category in the first category;

[0129] The second annotation unit 904 is used to call the specialized model to detect the instance boxes of the second category in the preset video image, and to annotate the instance boxes of the preset video image with pixel-level masks to obtain target image annotation data; wherein, the number of image frames of the preset video image is greater than a preset frame number threshold.

[0130] Furthermore, the acquisition unit 901 is specifically used to perform data cleaning on the original video to obtain cleaned video images; input the cleaned video images and instance information of the first category into a multimodal retrieval model to obtain video images containing the first category selected by the multimodal retrieval model from the cleaned video images.

[0131] Furthermore, the first annotation unit 902 is specifically used to input the video image and the category information of the first category into the multimodal large model to obtain the initial instance box of the first category in the video image detected by the multimodal large model with zero samples; and to perform deduplication processing on the initial instance box to obtain the instance box of the first category in the video image.

[0132] Furthermore, the video image includes: a series of video images in temporal sequence; the first annotation unit 902 is specifically used to: input the instance boxes of the series of video images into an image segmentation model; annotate the image pixels contained in the instance boxes of the series of video images with temporal masks based on the image segmentation model; remove the temporal masks in the series of video images whose overlap is greater than a preset overlap threshold and whose integrity is less than a preset integrity threshold, to obtain the initial image annotation data.

[0133] Furthermore, the training unit 903 is specifically used to: divide the initial image annotation data into a training set and a validation set in each training cycle; after training the initial specialized model on the training set, determine the first detection accuracy of the initial specialized model for the second category and the second detection accuracy of the multimodal large model for the second category on the validation set; compare the first detection accuracy with the second detection accuracy until the first detection accuracy is greater than the second detection accuracy, and obtain the trained specialized model.

[0134] Furthermore, the second annotation unit 904 is specifically used to call the multimodal large model to detect the instance boxes of long-tail categories in the preset video image, wherein the long-tail categories are categories other than the second category in the first category; and to annotate the instance boxes of the second category and the instance boxes of the long-tail categories in the preset video image with pixel-level masks to obtain target image annotation data.

[0135] Furthermore, the second annotation unit 904 is also configured to: when the amount of data of the target image annotation data is greater than a second quantity threshold, input any frame of video image from the preset video images into an expert model group to obtain the detection results of multiple models in the expert model group for the any frame of video image; the models in the expert model group are any one or both of the multimodal large model and the specialized model; determine the detection difficulty of the any frame of video image based on the error between the detection results of any two models in the expert model group for the any frame of video image; determine difficult video images from the preset video images whose detection difficulty is greater than a preset difficulty threshold; incrementally train the specialized model based on the image annotation data of the difficult video images, and call the incrementally trained specialized model for image annotation.

[0136] Furthermore, the detection result of the model in the expert model group for any frame of video image includes: the pixel-level mask corresponding to the instance box detected by the model in the expert model group for any frame of video image; the second annotation unit 904 is used to: use the pixel-level mask of one of the two models corresponding to any frame of video image as the prediction mask, and use the pixel-level mask of the other model corresponding to any frame of video image as the ground truth mask; calculate the intersection-over-union (IoU) index of any frame of video image based on the prediction mask and the ground truth mask; and determine the detection difficulty of any frame of video image based on the IoU index of any frame of video image.

[0137] This application also provides an electronic device 1000, such as... Figure 10 As shown, the electronic device 1000 of this application embodiment may include one or more central processing units (CPUs) 601 and a memory 1002, wherein the memory 1002 stores one or more applications or data.

[0138] The memory 1002 can be volatile or persistent storage. The program stored in the memory 1002 can include one or more modules, each module including a series of instruction operations on the electronic device. Furthermore, the central processing unit 1001 can be configured to communicate with the memory 1002 and execute the series of instruction operations stored in the memory 1002 on the electronic device 1000.

[0139] Electronic device 1000 may also include one or more power supplies 1005, one or more wired or wireless network interfaces 1004, one or more input / output interfaces 1003, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0140] The central processing unit 1001 can perform the operations performed by the first aspect or any specific method embodiment of the first aspect, which will not be described in detail here.

[0141] This application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described above.

[0142] This application also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to perform the method described above.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0147] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image annotation method, characterized in that, include: Obtain video images containing instances of a first category from the original video, where the first category contains multiple categories; The multimodal large model is invoked to detect the first category of instance boxes in the video image with zero samples, and the instance boxes in the video image are labeled with pixel-level masks to obtain the initial image annotation data; When the amount of initial image annotation data reaches a first quantity threshold, a specialized model with better instance detection accuracy for the second category than the multimodal large model is trained based on the initial image annotation data; wherein, the second category is any category in the first category; The specialization model is invoked to detect the second category of instance bounding boxes in the preset video image, and the instance bounding boxes of the preset video image are labeled with pixel-level masks to obtain target image annotation data; wherein, the number of image frames of the preset video image is greater than a preset frame number threshold.

2. The image annotation method according to claim 1, characterized in that, The step of obtaining video images containing instances of the first category from the original video includes: The original video is cleaned to obtain a cleaned video image; The cleaned video images and instance information of the first category are input into the multimodal retrieval model to obtain video images containing instances of the first category selected by the multimodal retrieval model from the cleaned video images.

3. The image annotation method according to claim 1, characterized in that, The step of calling a multimodal large model to detect instance boxes of the first category in the video image with zero shots includes: The video image and the category information of the first category are input into the multimodal large model to obtain the initial instance bounding box of the first category in the video image detected by the multimodal large model with zero samples; The initial instance boxes are deduplicated to obtain instance boxes of the first category in the video image.

4. The image annotation method according to claim 1, characterized in that, The video images include: multiple frames of video images that are sequentially continuous in time; The step of annotating the instance bounding boxes of the video image with pixel-level masks to obtain initial image annotation data includes: The instance bounding boxes of the multi-frame video images are input into the image segmentation model, and the image pixels contained in the instance bounding boxes of the multi-frame video images are labeled with temporal masks based on the image segmentation model. The initial image annotation data is obtained by removing the temporal masks in the multi-frame video images whose overlap is greater than a preset overlap threshold and whose integrity is less than a preset integrity threshold.

5. The image annotation method according to claim 1, characterized in that, The training of a specialized model based on the initial image annotation data, which achieves higher instance detection accuracy for the second category than the multimodal large model, includes: In each training cycle, the initial image annotation data is divided into a training set and a validation set; After training the initial specialized model on the training set, the first detection accuracy of the initial specialized model for the second category and the second detection accuracy of the multimodal large model for the second category are determined on the validation set. The first detection accuracy is compared with the second detection accuracy until the first detection accuracy is greater than the second detection accuracy, thus obtaining the trained specialized model.

6. The image annotation method according to claim 1, characterized in that, The step of annotating the instance bounding boxes of the preset video image with pixel-level masks to obtain target image annotation data includes: The multimodal large model is invoked to detect instance boxes of long-tail categories in the preset video image, wherein the long-tail category is a category other than the second category in the first category; Pixel-level masks are used to annotate the instance bounding boxes of the second category and the instance bounding boxes of the long-tail category in the preset video image to obtain target image annotation data.

7. The image annotation method according to claim 1, characterized in that, After annotating the instance bounding boxes of the preset video image with pixel-level masks to obtain the target image annotation data, the method further includes: When the amount of data of the target image annotation data is greater than the second quantity threshold, any frame of the video image in the preset video image is input into the expert model group to obtain the detection results of the multiple models in the expert model group for the any frame of the video image; the model in the expert model group is any one or two of the multimodal large model and the specialized model. The detection difficulty of any frame of video image is determined based on the error between the detection results of any two models in the expert model group for any frame of video image. From the preset video images, identify difficult video images whose detection difficulty is greater than a preset difficulty threshold; The specialized model is incrementally trained based on the image annotation data of the difficult example video images, and the incrementally trained specialized model is then used for image annotation.

8. The image annotation method according to claim 7, characterized in that, The detection results of the model in the expert model group for any frame of video image include: the pixel-level mask corresponding to the instance box detected by the model in the expert model group for any frame of video image; The determination of the detection difficulty of any given video frame based on the error between the detection results of any two models in the expert model group for any given video frame includes: Use the pixel-level mask of one of the two models corresponding to any frame of video image as the prediction mask, and use the pixel-level mask of the other model corresponding to any frame of video image as the ground mask. The intersection-over-union ratio (IoU) of any frame of video image is calculated based on the predicted mask and the real mask. The detection difficulty of any given video frame is determined based on the cross-union ratio (CUP) of that frame.

9. An image annotation device, characterized in that, include: The acquisition unit is used to acquire video images containing instances of a first category in the original video, wherein the first category contains multiple categories; The first annotation unit is used to call the multimodal large model to detect the instance boxes of the first category in the video image with zero samples, and to annotate the instance boxes of the video image with pixel-level masks to obtain the initial image annotation data; The training unit is used to train a specialized model with better instance detection accuracy for the second category than the multimodal large model based on the initial image annotation data when the amount of the initial image annotation data reaches a first quantity threshold; wherein, the second category is any category in the first category; The second annotation unit is used to call the specialized model to detect the instance boxes of the second category in the preset video image, and to annotate the instance boxes of the preset video image with pixel-level masks to obtain target image annotation data; wherein, the number of image frames of the preset video image is greater than a preset frame number threshold.

10. An electronic device, characterized in that, include: Processor, memory, input / output interfaces, wired or wireless network interfaces, power supply; The memory is either a short-term storage memory or a persistent storage memory; The processor is configured to communicate with the memory and execute instructions in the memory on a control plane functional entity to perform the method of any one of claims 1 to 8.

11. A computer program product comprising a computer program, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images