Method, device, equipment and product for annotating image

Abstract

The invention relates to a method, a device, equipment and a product for annotating an image. The method comprises the steps that existing annotation information is determined, the existing annotation information comprises content annotated by a user for images in a first subset in an image set, and the image set comprises a plurality of images in the same field. The method further comprises the step of generating corresponding semantic content through a language model based on the existing annotation information. The method further includes annotating images in a second subset of the set of images based on the semantic content. In this way, the annotation information contained after part of the images in the image set are annotated by the user can be used for assisting the annotation task of the other parts of the images, and therefore the annotation efficiency and accuracy can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more specifically, to methods, apparatus, devices, and products for annotating images. Background Technology

[0002] Labeling tasks refer to the process of marking or annotating different types of data (such as image data, video data, audio data, etc.). Image labeling involves marking objects or regions of interest in an image so that machine learning algorithms can recognize and classify these objects. The images that need to be labeled differ across different technological fields. For example, in the field of autonomous driving, images of the vehicle's surrounding environment can be labeled to construct training data rich in semantic information. This training data is crucial for improving the perception, decision-making, and safety capabilities of autonomous driving systems. Summary of the Invention

[0003] Embodiments of this disclosure provide a method, apparatus, device, and product for annotating images.

[0004] In a first aspect of this disclosure, a method for annotating images is provided. The method includes acquiring existing annotation information, which includes annotations by a user on images in a first subset of an image set, the image set comprising multiple images within the same domain. The method further includes determining corresponding semantic content based on the existing annotation information using a language model. The method also includes annotating images in a second subset of the image set based on the semantic content.

[0005] In a second aspect of this disclosure, an apparatus for annotating images is provided. The apparatus includes an annotation information acquisition unit configured to acquire existing annotation information, which includes content annotated by a user for images in a first subset of an image set. The apparatus also includes a semantic content generation unit configured to generate corresponding semantic content based on the existing annotation information using a language model. Finally, the apparatus includes an annotation unit configured to annotate images in a second subset of the image set based on the semantic content.

[0006] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method provided according to a first aspect of this disclosure.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions, which are executed by a processor to implement the method provided according to a first aspect of this disclosure.

[0008] In a fifth aspect of this disclosure, a computer program product is provided, which is tangibly stored on a non-volatile computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the method provided according to a first aspect of this disclosure.

[0009] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;

[0012] Figure 2 A flowchart of a method for annotating images according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A schematic diagram illustrating the flow of labeled images according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic diagram illustrating image annotation according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A schematic diagram showing annotation results according to some embodiments of the present disclosure is illustrated;

[0016] Figure 6 A block diagram of an apparatus for annotating images according to some embodiments of the present disclosure is shown; and

[0017] Figure 7 A block diagram of a device that can implement several embodiments of the present disclosure is shown. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] As mentioned above, annotation tasks can be completed manually or automatically using annotation models. Before using an annotation model to annotate images or other data, the model needs to be trained, but the training process is complex, and the resulting model has low robustness. Therefore, a basic annotation model trained on large-scale data can be used to annotate the data. This basic annotation model can accept multimodal data as input and achieve pixel-level prediction for accurate annotation. However, despite its strong generalization ability, when faced with domain-specific data, the semantics of that domain may deviate from the semantics of the training data (the data used to train the basic model), leading to lower-quality annotations from the basic model.

[0021] Therefore, embodiments of this disclosure provide a method for image annotation. This method includes utilizing existing annotations of some images within the same domain to assist in the annotation task of other images within that domain. For example, a language model can be used to generate semantic content based on existing annotations, and this semantic content can be used as reference information to automatically annotate other images within the domain. By incorporating user-generated annotations for a specific domain image dataset and combining them with semantic content generated by a language model, the semantic features of the domain corresponding to the image can be more accurately understood and captured during image annotation, thereby improving the accuracy and reliability of the annotation.

[0022] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of this disclosure may be implemented is shown. For example... Figure 1As shown, environment 100 includes an image set 102 and an electronic device 108 for image annotation. Electronic device 108 can be any electronic device capable of executing the methods according to this disclosure. For example, electronic device 108 can be an electronic device capable of running a trained neural network model to execute the methods according to this disclosure. For example, electronic device 108 can be a user terminal, mobile device, computer, or vehicle terminal, or it can be a computing system, a single server, a distributed server, or a cloud-based server. Electronic device 108 may be equipped with a language model 110 and an annotation model 114. The annotation model 114 can be a basic annotation model trained using large-scale training data. The language model 110 can include various types, such as statistical language models, neural network language models (e.g., convolutional neural networks, recurrent neural networks), etc. In some examples, the language model 110 can be a pre-trained language model, such as a GPT model, a BERT model, etc.

[0023] It is understood that the image annotation method provided in the embodiments of this disclosure can be applied to a variety of fields and application scenarios, such as autonomous driving, medical image analysis, remote sensing image analysis, security monitoring, and industrial automation.

[0024] According to embodiments of this disclosure, in example environment 100, image set 102 can be acquired in real time by an image acquisition device, or it can be obtained from an open-source training dataset. For example, when the image annotation method provided in embodiments of this disclosure is applied to the field of autonomous driving, image set 102 can be acquired by sensors deployed on a vehicle. Sensors can include, but are not limited to, vision sensors (also known as optical cameras or video cameras), millimeter-wave radar, lidar, global positioning systems, ultrasonic sensors, inertial measurement units, and other sensors. Sensors deployed on a vehicle can acquire multiple images of the vehicle's internal or external environment. It is understood that multiple images can be images of the same scene or the same domain (e.g., multiple images belonging to the field of autonomous driving), and image set 102 can be constructed based on multiple images.

[0025] In some embodiments of this disclosure, to assist the basic annotation model 114 in more accurately annotating the images in image set 102, a user can annotate a portion of the images in image set 102 (e.g., image set 104 within image set 102). For example, the user can annotate the first n images in image set 102 to determine existing annotation content. That is, image set 104 contains multiple images and their corresponding existing annotation content. The existing annotation content may include the location and category of objects contained in the images already annotated by the user. The multiple images in image set 104 and the existing annotation information contained in those images can be input into language model 110.

[0026] Understandably, to enable the annotation model 114 to better understand the annotation task and achieve higher annotation accuracy, the semantics within a specific domain (such as the semantics contained in existing annotation content) can be aligned with the semantics that the base model can understand. For example, the language model 110 can generate semantic content 112 corresponding to existing annotation information. The language model 110 performs semantic interpretation or semantic expansion on the existing annotation information to generate the corresponding semantic content 112. In one example, the existing annotation information may include "human.pedestrian.stroller". The semantic content generated by the language model 110 can be "A stroller is a lightweight vehicle designed for the convenient transportation of infants and young children. A stroller refers to an individual pushing a stroller or baby carriage, usually used to transport infants or young children. Strollers generally have four small wheels, making them easy for parents or caregivers to push and operate."

[0027] In some embodiments of this disclosure, the semantic content 112 determined by the language model 110 can be input into the annotation model 114 to assist the annotation task of the annotation model 114 (e.g., to assist the annotation model 114 in the annotation task of image set 106 in the image set). It is understood that the semantic content 112 is natural semantic content that can be understood by the annotation model 114, such as semantic content composed of strings of natural semantics. That is, the semantic content 112 can help the annotation model 114 better understand the annotation task and eliminate semantic biases for the annotation model 114. The annotation model 114 can annotate the images in the input image set 106 (e.g., images in image set 102 that have not been annotated by the user) based on the semantic content to determine the corresponding annotation content 116. For example, the annotation model 114 may include a detection model and a segmentation model. The detection model is used to annotate objects in the image and their categories using two-dimensional bounding boxes. The segmentation model is used to perform semantic segmentation or instance segmentation on the image, determining the category corresponding to each pixel in the image and a semantic mask composed of multiple pixels of the same category. The annotation content 116 may include two-dimensional annotation boxes for objects in the image, object category labels, pixel category labels, etc.

[0028] It is understood that the image annotation method provided in this embodiment can also be used to annotate point cloud data or other data. For example, the annotation information of a portion of the point cloud in the 3D point cloud data can be used to assist in the annotation task of the remaining point cloud data in the 3D point cloud data, thereby improving the annotation efficiency and accuracy of the 3D point cloud data.

[0029] By incorporating user-annotated image datasets for specific domains and combining them with semantic content extracted by pre-trained language models, the annotation model can more accurately understand and capture semantic features of specific domains. This improves the accuracy of the basic annotation model on specific domain data, further enhances the model's generalization ability on specific domain data, and makes it better adaptable to application scenarios in different domains.

[0030] The following will combine Figures 2 to 7 The process according to embodiments of this disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0031] Figure 2 A flowchart of a method 200 for annotating an image according to some embodiments of the present disclosure is shown. Method 200 can be... Figure 1 The electronic device 108 performs this action. For example... Figure 2 As shown in box 202, method 200 includes obtaining existing annotation information, which includes user-defined annotations for an image set (e.g., Figure 1 The annotations refer to the images in the first subset of the image set 102 shown (e.g., images in image set 104), which include multiple images within the same domain (e.g., multiple images within the domain of autonomous driving). The image set may include multiple images for the same scene or within the same domain. It is understood that the images in the first subset refer to images annotated by the user. In one example, the image set may include 1000 images, and the first subset may include the first 300 images annotated by the user.

[0032] In some embodiments of this disclosure, the image set may include multiple images acquired by the vehicle's sensors at a preset frame rate. User annotations may include bounding boxes (e.g., two-dimensional or three-dimensional bounding boxes) of objects in the images, object category labels, or pixel categories, etc. For example, a user can use rectangles to annotate objects contained in an image and add category labels to the rectangles. Category labels refer to the category to which an object in an image belongs. Category labels differ across different fields; for example, in the field of autonomous driving, category labels may include "pedestrian," "vehicle," "lane line," "traffic sign," etc.

[0033] In box 204, method 200 includes, based on existing annotation information, using a language model (e.g., Figure 1 The language model 110 shown generates corresponding semantic content. Here, the language model refers to an artificial intelligence model trained on a large-scale corpus. These models can understand and generate natural language text and demonstrate excellent performance in various natural language processing tasks. For example, tasks may include, but are not limited to, text classification, named entity recognition, sentiment analysis, and machine translation. In some embodiments of this disclosure, pre-training can be performed on large-scale text data, enabling the language model to learn general language representations. After pre-training, fine-tuning can be used to adapt it to specific tasks or domains.

[0034] It's understandable that objects in images within a specific domain vary, leading to diverse category labels in existing annotation information. Furthermore, these domain-specific category labels also have different meanings (e.g., different domains have unique terms, concepts, or expressions), generally incomprehensible to basic annotation models. To leverage existing annotation models for annotation tasks without requiring specially trained models, domain-specific semantics can be matched and transformed with general semantics in the network, bridging the semantic gap between specific and general domains. This allows the basic annotation model to understand the annotation task and complete it accurately and efficiently. For example, language models can be used to interpret and expand existing annotation information, obtaining natural semantics that the annotation model can understand, making the annotation results more consistent and reliable. The annotation model can be an existing basic annotation model trained on large-scale network data, possessing strong generalization ability without being domain-specific.

[0035] In box 206, method 200 includes, based on semantic content, applying the second subset of the image set (e.g., Figure 1 The images in the image set 106 shown are labeled. For example, the remaining images in the image set can be labeled using a basic open-source labeling model. For example, bounding boxes (BBoxes) can be used to label objects in the images, and category labels can be added to the objects. In some embodiments of this disclosure, the labeling model can also perform instance segmentation or semantic segmentation on the input image, determine the category of each pixel in the image, and determine the semantic mask corresponding to each object in the image. In some embodiments of this disclosure, the labeling model can determine the region or object in the image that corresponds to the semantic content based on the semantic content. It can be understood that the labeling content corresponding to the labeled image may include the label boxes corresponding to the objects in the image and the categories, or it may include the categories of each pixel in the image and semantic boxes composed of pixels of the same category.

[0036] In this way, the language model interprets the existing annotation information, thereby aligning the semantics of different domains with the semantics of the network. This allows the annotation model to better understand the image content, thereby improving the efficiency and accuracy of data annotation and better meeting the needs of annotation tasks in different domains.

[0037] In some embodiments of this disclosure, the category labels can be hierarchical. For example, the "vehicle" level can be further subdivided into "sedan," "truck," "bus," etc., and the "sedan" level can be further subdivided into "large sedan," "medium sedan," "electric vehicle," etc. It is understood that the hierarchy can be set by the annotators. When setting the hierarchy, it should be reasonably divided according to the complexity of the task and the diversity of objects to ensure that subsequent annotation tasks can effectively identify and understand the objects in the image.

[0038] In some embodiments of this disclosure, the annotation model may be a basic annotation model trained using general training data. For example, the annotation model may be a deep learning-based annotation model. Annotation model (e.g. Figure 1 The annotation model 114 shown can determine the region or object in the image that corresponds to the semantic content of the input. The annotation model can use two-dimensional bounding boxes to annotate the region or object corresponding to the semantic content, and use a segmentation model to perform semantic segmentation on the image to determine the category of each pixel in the image. In one example, the annotation model can include a GroundingDINO model and a semantic segmentation model (e.g., a Segment Anything Model, SAM). The GroundingDINO model can determine matching image features from the image based on the semantic content, while the SAM model can determine the region or object corresponding to the image features from the input image. For example, the GroundingDINO model can identify the objects contained in the input image using bounding boxes.

[0039] In some embodiments of this disclosure, since the annotation model is an open-source basic annotation model, it is usually trained using general, non-domain-specific datasets and may not be able to directly understand the semantics within a specific domain (e.g., it may not understand the terms, definitions, etc. within a specific domain). For example, the annotation model is unclear about and cannot understand the semantics of the label "human.pedestrian.stroller" in the field of autonomous driving. Directly annotating images in an image set using the annotation model is inaccurate. Therefore, the semantic interpretation can be performed on the labeling system that has been established based on the user's annotation information, so that the determined semantic content can be understood by the annotation model and aligned with the semantics that the annotation model can understand.

[0040] In some embodiments of this disclosure, a language model can be used to semantically interpret existing annotation information. For example, category labels (such as "pedestrian," "vehicle," "building," "traffic light," etc.) contained in existing annotation information can be input into a pre-trained language model, which then generates a natural language description related to the category label. The natural language description related to the category label can be a synonym, near-synonym, or extended word of the category label, or a more detailed and easier-to-understand explanation of the category label. Synonyms can be other words with similar meanings to the category. For example, synonyms for "car" might include "four-wheel drive," "sedan," "electric car," etc. Extended words are broader or more specific concepts that may be related to the category label. For example, for the category label "pedestrian," extended words might include "walker," "person crossing the road," "person waiting for a traffic light," etc. These extended words can provide more specific contextual information, helping the annotation model understand the specific scene in the image.

[0041] In this way, the semantics within a specific domain can be aligned with the semantics that the basic open-source annotation model can understand, making the annotation results of the annotation model more accurate while better meeting the annotation needs of different domains.

[0042] In some embodiments of this disclosure, to further improve the accuracy of the annotation results, multiple images from the image set can be input into a language model or other pre-trained model, causing the model to output natural language describing the image features. In some embodiments, image features can be extracted using image recognition or other methods and converted into natural language descriptions. For example, for an image containing pedestrians and vehicles, the natural language description of the image features might include "There is a pedestrian and a car in the image." It is understood that the natural language and semantic content used to characterize image features can serve as auxiliary information to assist the annotation model in completing the annotation task.

[0043] In some embodiments of this disclosure, other input information can be combined to assist the annotation task of the annotation model. The input information can be 3D point cloud data corresponding to the image, or the 3D bounding box of the object in the image within the 3D point cloud data. The 3D bounding box can be projected to obtain a corresponding 2D box. For example, the 3D bounding box can be projected onto the camera coordinate system, and the corner points (e.g., 8 corner points) of the 3D bounding box can be projected onto the image plane. Multiple corner points can be selected to obtain the 2D box (e.g., selecting the 4 largest corner points). It can be understood that the 2D box is a rough 2D box, which can be larger or smaller than the actual size of the object in the image. Most of the area of ​​the object in the image can lie within the 2D box.

[0044] In some embodiments of this disclosure, an image, semantic content, and two-dimensional bounding boxes can be input into an annotation model, which then annotates the image based on the semantic content and the bounding boxes. The bounding boxes assist the annotation model by providing a rough representation of the location of objects in the image. For example, the annotation model can refine or modify the rough bounding boxes based on the semantic content to obtain the final two-dimensional bounding boxes. In one example, the GroundingDINO model can identify the input image based on the semantic content and mark the objects contained in the image with rectangular bounding boxes, obtaining corresponding candidate two-dimensional bounding boxes. The two-dimensional bounding boxes are then refined or modified based on the candidate bounding boxes to obtain the final two-dimensional bounding boxes.

[0045] In some embodiments of this disclosure, the three-dimensional point cloud data can be further combined to determine a more accurate annotation result. For example, the three-dimensional point cloud data can be projected onto an image plane to obtain corresponding two-dimensional point cloud data. Alternatively, points located within three-dimensional bounding boxes can be extracted from the three-dimensional point cloud data, and these points can be projected onto an image plane to obtain two-dimensional points in the image. It is understood that two-dimensional points and two-dimensional boxes can be used to assist the annotation model in annotating the input image to obtain corresponding two-dimensional annotation boxes. In other embodiments of this disclosure, the three-dimensional point cloud data contained in the three-dimensional bounding boxes can be sampled to obtain a preset number of three-dimensional points (e.g., 3-5 points). In some embodiments of this disclosure, the sampling method can be random sampling, farthest point sampling (FPS), or other sampling methods.

[0046] In some embodiments of this disclosure, a preset number of three-dimensional points can be projected onto an image plane to obtain a preset number of two-dimensional points. These preset number of two-dimensional points are the pixels corresponding to objects in the image and can be used as annotation points. It can be understood that, based on these annotation points and the two-dimensional bounding boxes, the semantic segmentation model in the annotation model can be used to perform semantic segmentation on the image, determining the category corresponding to each pixel in the image and a semantic mask composed of multiple pixels of the same category.

[0047] In some embodiments of this disclosure, to ensure more accurate output annotation results, a quality assessment can be performed on the annotated content after annotating the input image to determine the annotation quality corresponding to the annotated content. Annotation quality can be a specific quality score (e.g., a score within the range of 0-100) or a quality level (e.g., high quality, low quality, etc.). It can be understood that the annotation content includes two aspects: geometric aspects (e.g., the size of the annotation box, geometric features) and semantic aspects (e.g., the category of the object, the category of pixels, or the semantic box or semantic mask of the object). Based on this, the annotation quality of the annotated content can be determined from two aspects (semantic quality assessment and geometric quality assessment). For example, a semantic quality score can be determined based on semantic features, and a geometric quality score can be determined based on geometric features. The annotation quality score can be the sum of the two, a weighted sum of the two, or a weighted average of the two.

[0048] In some embodiments of this disclosure, annotation quality can be determined based on the difference between the size of the annotation frame and the conventional size of the object. If the size of the annotation frame differs significantly from the conventional size of the object, or if the aspect ratio of the annotation frame differs significantly from a preset aspect ratio, the annotation frame is determined to be inaccurate, and the annotation quality is low. For example, in one example, the conventional length of a small car is generally 3.8m-5m, and the width is generally 1.5m-2m. If the length of the annotation frame exceeds 10m, it indicates that the annotation frame is inaccurate, the annotation quality is low, and the annotation quality score can be 30 points (far less than the preset score threshold of 80 points).

[0049] In other embodiments of this disclosure, annotation quality can also be determined based on the geometric continuity of the semantic mask of the object. For example, if the semantic mask of a vehicle contains multiple holes and is not continuous, the semantic segmentation result can be determined to be inaccurate and the annotation quality to be low. Or, if the edges of the semantic mask are incomplete or have unreasonable overlap with other semantic masks, the annotation quality can be determined to be low. In one example, if most pixels in the same region are classified as vehicles, while a small portion of pixels are classified as sky, it indicates that the segmentation result of these small portion of pixels is inaccurate.

[0050] In some embodiments of this disclosure, after segmenting an image using a segmentation model, a semantic similarity model (e.g., a model combining visual and textual information) can be used to determine the corresponding semantic similarity. A high semantic similarity indicates that the semantic information of the semantic mask is correct; a low semantic similarity indicates that the semantic segmentation model has incorrect segmentation results. For example, a semantic similarity model can receive a semantic mask (or a region of the input image overlaid with the mask) and one or more possible labels (or descriptive text), and then output a score representing the semantic similarity between the semantic mask and the given label. It can be understood that if the semantic similarity is less than a preset similarity threshold, the annotation quality can be determined to be low; if the semantic similarity is greater than the preset similarity threshold, the annotation quality can be determined to be high.

[0051] In some embodiments of this disclosure, if the annotation quality does not meet preset requirements (e.g., the annotation quality score is less than a preset score threshold), the annotation process is repeated (e.g., the semantic content corresponding to the existing annotation information is redefined, the 2D bounding box corresponding to the 3D bounding box is redefined, the annotation model is re-applied to the image, etc.). If the annotation quality after re-annotation meets the preset requirements, the corresponding annotation result is output. If the annotation quality after re-annotation does not meet the preset requirements, the annotation process is repeated again until the number of re-annotations exceeds a preset threshold. If the determined annotation quality still does not meet the preset requirements when the number of re-annotations exceeds the preset threshold, the annotation work for the image is abandoned.

[0052] Figure 3 A schematic diagram illustrating the flow of annotated images according to some embodiments of the present disclosure is shown. For example... Figure 3 As shown, the example environment 300 includes input information 308, a prompt processing module 310, an annotation module 318, and an annotation quality determination module 324. It can be understood that the prompt processing module 310 may contain multiple pre-trained models, such as pre-trained language models.

[0053] In some embodiments of this disclosure, input information 308 includes various types of input, such as other inputs 302 (e.g., 3D point cloud data, 3D bounding boxes, etc.), text prompts 304, and input images 306. It can be understood that 3D point cloud data is point cloud data associated with the input image 306. For example, the 3D point cloud data and image 306 were acquired for the same scene within the same time period. The 3D bounding box can be the 3D bounding box corresponding to the object contained in image 306. In some embodiments of this disclosure, text prompts 304 can be content annotated by the user for a portion of images in the image set, such as annotated category labels.

[0054] like Figure 3As shown, the prompt processing module 310 may include multiple processing modules, such as a semantic prompt word processing module 312, a bounding box prompt word processing module 314, and a point prompt word processing module 316. The semantic prompt word processing module 312 can use a pre-trained language model to perform semantic interpretation and semantic expansion on the text prompt 304, generating corresponding semantic content (not shown in the figure). The bounding box prompt word processing module 314 can project the 3D bounding box to determine the corresponding 2D box (not shown in the figure). The point prompt word processing module 316 can project the 3D point cloud data to determine the corresponding 2D point (not shown in the figure).

[0055] In some embodiments of this disclosure, two-dimensional points, two-dimensional bounding boxes, and semantic content can all serve as auxiliary reference information for the annotation model to annotate images. It is understood that all three can be input into the annotation module 318 as reference information for the annotation model, or only some information can be input into the annotation module 318. In one embodiment of this disclosure, a two-dimensional bounding box can be input into the annotation module 318, which then annotates the input image 306 based on the bounding box to obtain the annotation result.

[0056] In some embodiments of this disclosure, the annotation module 318 may include a detection model 320 and a segmentation model 322. The detection model 320 (e.g., the GroundingDINO model) can use two-dimensional bounding boxes to annotate objects in the image 306 and assign category labels to the bounding boxes. The segmentation model 322 can perform instance segmentation or semantic segmentation on the image 306 based on the bounding boxes or the two-dimensional bounding boxes, determining the category corresponding to each pixel in the image. Based on the segmentation results, multiple semantic masks corresponding to multiple objects in the image 306 can be determined.

[0057] It is understood that the annotation results can be input to the annotation quality determination module 324, which determines the annotation quality of the results. The annotation quality determination module 324 may include a geometric quality determination module 326 and a semantic quality determination module 328. The geometric quality determination module 326 determines the geometric quality based on the geometric features of the two-dimensional bounding box or the geometric features of the semantic mask. The semantic quality determination module 328 determines the semantic quality based on the semantic features of the two-dimensional bounding box or the semantic features of the semantic mask. Based on the geometric and semantic quality, the annotation quality corresponding to the annotation result can be determined. If the annotation quality does not meet the preset requirements, the input information can be reprocessed, and the annotation work for image 306 can be redone. The annotation quality can be determined based on the semantic segmentation results and object detection results contained in the annotation content. The semantic segmentation results may include pixel category labels, semantic similarity, and semantic masks. The object detection results may include the object's location and category. It is understandable that if the quality of the re-annotation meets the preset requirements, the annotation content 330 will be output; if the quality of the annotation after multiple re-annotations still does not meet the preset requirements, the annotation work of image 306 will be abandoned.

[0058] Figure 4 A schematic diagram illustrating image annotation according to some embodiments of the present disclosure is shown. For example... Figure 4 As shown, the existing annotation information 402 can be input into the pre-trained language model 406, which determines the corresponding semantic content. The 3D bounding box 404 can be input into the point cloud processor 408, which determines the corresponding 2D bounding box. The semantic content, 2D bounding box, and image 410 can be input into the annotation model 412. The annotation model 412 can annotate the input image 410 based on the semantic content and 2D bounding box, determining the corresponding annotation content. For example, the annotation model 412 can use the 2D bounding box to annotate objects in the image and determine the category label of the object. The annotation model 412 can also perform semantic segmentation on the image and determine the corresponding semantic segmentation result.

[0059] In some embodiments of this disclosure, the labeled content can be input to the labeling quality evaluation module 414, which determines the corresponding labeling quality. For example, the labeling quality evaluation module 414 can determine the corresponding labeling quality 416 based on the geometric features of the bounding box and the semantic segmentation results. If the labeling quality 416 does not meet the requirements, re-labeling is performed. For example, the pre-trained language model 406 re-interprets and expands the existing labeled information 402 to determine new semantic content. If the labeling quality after re-labeling still does not meet the requirements, the labeling process can be repeated again. If the number of repeated labelings exceeds the labeling count threshold and the labeling quality still does not meet the requirements, the labeling work for the image is abandoned. It should be noted that the above embodiments are only examples illustrating one labeling process. Other labeling processes may also exist in other embodiments, and the input information used in other labeling processes may be the same or different.

[0060] Figure 5 A schematic diagram illustrating annotation results according to some embodiments of the present disclosure is shown. For example... Figure 5 As shown, the annotation results for image 502 can include bounding boxes (dashed boxes) for objects in image 502 and category labels for the objects. For example, objects in image 502 can include "pedestrians," "buildings," "trees," and "vehicles." After annotating the objects in image 502 with bounding boxes, category labels can be added to the bounding boxes. For example, the category label for the vehicle bounding box can be "car," and the category label for the pedestrian bounding box can be "people."

[0061] Figure 6 A block diagram of an apparatus 600 for annotating images according to some embodiments of the present disclosure is shown. Figure 6 As shown, the device 600 includes an annotation information acquisition unit 602, configured to acquire existing annotation information, which includes content annotated by the user for images in a first subset of the image set. The device 600 also includes a semantic content generation unit 604, configured to generate corresponding semantic content based on the existing annotation information using a language model. The device 600 further includes an annotation unit 606, configured to annotate images in a second subset of the image set based on the semantic content.

[0062] In some embodiments, the semantic content generation unit 604 is further configured to: acquire multiple category labels contained in existing annotation information; determine content for interpreting the multiple category labels through a language model; and generate the semantic content based on the content for interpreting the multiple category labels and natural language for describing the image features corresponding to the images in the second subset.

[0063] In some embodiments, the semantic content generation unit 604 is further configured to generate content for explaining the multiple category tags based on multiple synonyms and multiple extended words of multiple category tags.

[0064] In some embodiments, the annotation unit 606 is further configured to: determine the two-dimensional bounding boxes corresponding to the projection of the three-dimensional bounding boxes based on the three-dimensional bounding boxes corresponding to the objects contained in the images in the second subset, wherein the three-dimensional bounding boxes are bounding boxes in the three-dimensional point cloud data corresponding to the images in the second subset; and annotate the images in the second subset of the image set based on the semantic content and the two-dimensional bounding boxes.

[0065] In some embodiments, the annotation unit 606 is further configured to: determine multiple two-dimensional position information corresponding to multiple corner points of the three-dimensional bounding box by projecting the three-dimensional bounding box onto the image plane; determine multiple target corner points based on the multiple two-dimensional position information corresponding to the multiple corner points; and determine a two-dimensional box corresponding to the three-dimensional bounding box based on the multiple target corner points.

[0066] In some embodiments, the annotation unit 606 is further configured to: determine two-dimensional candidate annotation boxes for objects contained in the images of the second subset based on the semantic content; and determine two-dimensional annotation boxes for the objects based on the two-dimensional candidate annotation boxes and the two-dimensional boxes.

[0067] In some embodiments, the annotation unit 606 is further configured to: determine three-dimensional target point cloud data by sampling the three-dimensional point cloud data contained in the three-dimensional bounding box; and determine the two-dimensional pixel points corresponding to the object by projecting the three-dimensional target point cloud data onto the image plane.

[0068] In some embodiments, the labeled content of the image includes the segmentation result corresponding to the image, and the apparatus 600 further includes a labeling quality score determination unit, configured to: determine a semantic box for an object based on the segmentation result; and determine the labeling quality score of the labeled content based on the semantic box and the geometric and semantic features of the two-dimensional label box.

[0069] In some embodiments, the geometric features include whether the semantic box is continuous and the size of the two-dimensional annotation box, and the annotation quality score determination unit is further configured to determine a first annotation quality score of the annotation content based on whether the semantic box is continuous and whether the size of the two-dimensional annotation box matches the preset size of the object.

[0070] In some embodiments, the annotation quality score determination unit is further configured to: determine a second annotation quality score of the annotation content based on the similarity between semantic features and semantic content.

[0071] In some embodiments, the annotation quality score determination unit is further configured to: determine the annotation quality score of the annotation content based on the first annotation quality score and the second annotation quality score.

[0072] In some embodiments, the apparatus 600 further includes a labeling quality score determination unit, configured to: in response to a labeling quality score being less than a preset quality score threshold, re-label the image and determine the labeling quality score after re-labeling, until the number of re-labelings exceeds a preset number threshold.

[0073] It is understood that by utilizing the apparatus 600 of this disclosure, at least one of the many advantages achievable by the methods or processes described above can be realized. For example, the apparatus 600 is capable of marking quality and marking accuracy.

[0074] Figure 7 A schematic block diagram of an example device 700 that can be used to implement embodiments of the present disclosure is shown. Figure 7 As shown, device 700 includes a processor 701, which can perform various appropriate actions and processes based on computer program instructions loaded into random access memory (RAM) 703 according to computer program instructions stored in read-only memory (ROM) 702. RAM 703 may also store various programs and data required for the operation of device 700. The processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0075] The various processes and procedures described above, such as method 200, can be executed by processor 701. For example, in some embodiments, method 200 may be implemented as a computer software program tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702. When the computer program is loaded into RAM 703 and executed by processor 701, one or more actions of method 200 described above may be performed.

[0076] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0077] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), and any suitable combination thereof. The computer-readable storage medium as used herein is not to be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0078] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0079] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0080] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0081] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0084] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method (200) for annotating images, comprising: obtaining (202) existing annotation information, the existing annotation information comprising content annotated by a user for images in a first subset of a set of images, the set of images comprising a plurality of images in a same domain; generating (204) corresponding semantic content by a language model based on the existing annotation information; and annotating (206) images in a second subset of the set of images based on the semantic content.

2. The method (200) of claim 1, wherein generating (204) corresponding semantic content by a language model comprises: obtaining a plurality of category labels contained in the existing annotation information; generating content for explaining the plurality of category labels by the language model; and generating the semantic content based on the content for explaining the plurality of category labels and natural language for describing corresponding image features of the images in the second subset.

3. The method (200) of claim 2, wherein generating content for explaining the plurality of category labels comprises: generating content for explaining the plurality of category labels based on a plurality of synonyms and / or a plurality of expansions of the plurality of category labels.

4. The method (200) of claim 1, wherein annotating (206) images in a second subset of the set of images comprises: determining corresponding two-dimensional boxes of three-dimensional bounding boxes contained in objects of the images in the second subset based on the three-dimensional bounding boxes, the three-dimensional bounding boxes being bounding boxes in three-dimensional point cloud data corresponding to the images in the second subset; and annotating the images in the second subset of the set of images based on the semantic content and the two-dimensional boxes.

5. The method (200) of claim 4, wherein determining corresponding two-dimensional boxes of the three-dimensional bounding boxes comprises: determining a plurality of two-dimensional position information of a plurality of corner points of the three-dimensional bounding boxes by projecting the three-dimensional bounding boxes onto an image plane; determining a plurality of target corner points based on the plurality of two-dimensional position information of the plurality of corner points; and determining the two-dimensional boxes corresponding to the three-dimensional bounding boxes based on the plurality of target corner points.

6. The method (200) of claim 4, wherein annotating (206) images in a second subset of the set of images comprises: determining two-dimensional candidate annotation boxes of objects contained in the images in the second subset based on the semantic content; and determining two-dimensional annotation boxes for the objects based on the two-dimensional candidate annotation boxes and the two-dimensional boxes.

7. The method (200) of claim 6, wherein annotating (206) images in a second subset of the set of images further comprises: determining three-dimensional target point cloud data by sampling three-dimensional point cloud data contained in the three-dimensional bounding boxes; and determining two-dimensional pixel points corresponding to the objects by projecting the three-dimensional target point cloud data onto an image plane. ​ ​ ​ ​ ​ ​ 8. The method (200) of claim 6, wherein the annotation content of the images in the second subset comprises segmentation results corresponding to the images, and the method (200) further comprises: determining semantic boxes for the objects based on the segmentation results; and determining an annotation quality score of the annotation content based on the semantic boxes and respective geometric features and semantic features of the two-dimensional annotation boxes.

9. The method (200) of claim 8, wherein the geometric features comprise whether the semantic boxes are continuous, sizes of the two-dimensional annotation boxes, and determining the annotation quality score of the annotation content comprises: determining a first annotation quality score of the annotation content based on whether the semantic boxes are continuous, whether sizes of the two-dimensional annotation boxes match preset sizes of the objects.

10. The method (200) of claim 9, wherein determining the annotation quality score of the annotation content comprises: determining a second annotation quality score of the annotation content according to a similarity between the semantic features and the semantic content.

11. The method (200) of claim 10, wherein determining the annotation quality score of the annotation content comprises: determining the annotation quality score of the annotation content based on the first annotation quality score and the second annotation quality score.

12. The method (200) of claim 8, further comprising: responsive to the annotation quality score being less than a preset quality score threshold, re-annotating the images and determining a re-annotation quality score until a number of re-annotations is greater than a preset number threshold.

13. An apparatus (600) for annotating images, comprising: an annotation information obtaining unit (602) configured to obtain existing annotation information, the existing annotation information comprising content annotated by a user for images in a first subset of a set of images, the set of images comprising a plurality of images in a same domain; a semantic content generating unit (604) configured to generate corresponding semantic content based on the existing annotation information by a language model; and an annotation unit (606) configured to annotate images in a second subset of the set of images based on the semantic content.

14. An electronic device (700), comprising: at least one processor (701); and a memory (702) coupled to the at least one processor (701) and having stored therein instructions that, when executed by the at least one processor (701), cause the device (700) to perform the method of any one of claims 1-12.

15. A computer program product tangibly stored on a non-transitory computer- readable medium and comprising machine executable instructions that, when executed, cause a machine to perform the steps of the method of any one of claims 1-12. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images