Image annotation method and device, medium and equipment
By expanding and automatically annotating the initial image dataset, combined with virtual 3D rendering and cross-validation, the problem of low image annotation efficiency was solved, and an efficient and accurate image annotation process was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING SHOUGANG AUTOMATION INFORMATION TECH
- Filing Date
- 2025-12-24
- Publication Date
- 2026-04-17
AI Technical Summary
Current image annotation processes are inefficient, time-consuming, labor-intensive, and costly, making it difficult to efficiently annotate image datasets.
By augmenting the initial image dataset, augmented images are generated using a virtual 3D space and rendering engine, and automatically labeled using a pre-trained visual cue encoder and cross-attention mechanism. Cross-validation is then performed to ensure labeling quality.
It has significantly improved the efficiency of image annotation, reduced the time and cost of data collection and the workload of annotation, and ensured that the annotation quality reaches more than 99%.
Smart Images
Figure CN121884033A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image data processing technology, and in particular to an image annotation method, apparatus, medium and device. Background Technology
[0002] With the development of deep learning technology, the performance of algorithm models largely depends on large-scale, high-quality labeled datasets. The higher the quality of the labeled dataset, the better the prediction accuracy of the algorithm model after subsequent training.
[0003] For example, when it comes to image annotation datasets, the image data annotation process in related technologies is highly dependent on manual labor. Annotators need to draw bounding boxes, segmentation masks, or assign category labels to each image one by one. This method is not only inefficient and time-consuming, but also has high labor costs.
[0004] Therefore, improving the efficiency of image annotation is an urgent problem to be solved. Summary of the Invention
[0005] To address the problems existing in the prior art, embodiments of the present invention provide an image annotation method, apparatus, medium, and device to solve or partially solve the technical problem that image annotation efficiency cannot be guaranteed in the prior art.
[0006] A first aspect of the present invention provides an image annotation method, the method comprising:
[0007] The initial image dataset is expanded to obtain an expanded image dataset; the initial image dataset is a labeled dataset.
[0008] The augmented image dataset is pre-annotated based on the initial image dataset to obtain the pre-annotation result for each augmented image;
[0009] For each pre-annotated augmented image in the augmented image dataset, the pre-annotation results of the augmented image are reviewed to obtain m reviewed annotation results for each augmented image; the m reviewed annotation results of each augmented image are cross-validated, and if the verification results meet the requirements, the augmented image is determined to be successfully labeled; m is greater than or equal to 2 and is a positive integer.
[0010] In the above scheme, expanding the initial image dataset includes:
[0011] Based on each initial image in the initial image dataset, a target image is searched in the historical sample database; the target image is an image whose similarity to the initial image meets a preset similarity threshold.
[0012] The target image is identified as an augmented image.
[0013] In the above scheme, expanding the initial image dataset includes:
[0014] Convert the labeled target objects in the initial image into three-dimensional models;
[0015] Within a virtual 3D space, based on the parameters of the virtual camera and the lighting conditions, a rendering engine is used to render the 3D model of the target object to obtain a composite image.
[0016] The synthesized image is identified as an augmented image.
[0017] In the above scheme, the step of annotating the expanded image dataset based on the initial image dataset includes:
[0018] Obtain each initial image from the initial image dataset and the first mask image of each initial image;
[0019] The aggregated feature vector of all initial images is determined based on each initial image and the first mask image of each initial image;
[0020] Each augmented image in the augmented image dataset is converted into a corresponding feature map;
[0021] For each combination of aggregated feature vector and feature map, the aggregated feature vector in the combination is used to perform context matching on the feature map to obtain an enhanced image of the feature map;
[0022] The enhanced image is decoded to obtain the second mask image corresponding to the enhanced image and the probability of each region in the second mask image. Regions with a probability greater than a preset probability threshold are determined as target regions.
[0023] The target region is labeled with its type based on the label type in the aggregated feature vector, and a label box is drawn for the target region.
[0024] In the above scheme, determining the aggregated feature vector based on each initial image and the first mask image of each initial image includes:
[0025] For each initial image, the first mask image of the initial image is superimposed on the initial image to obtain a visual cue image;
[0026] Each visual cue image is encoded using a pre-trained visual cue encoder to obtain the feature vector corresponding to each visual cue image;
[0027] According to the formula The feature vectors corresponding to each of the aforementioned visual cue images are aggregated to obtain an aggregated feature vector P. store ;in,
[0028] n is the total number of feature vectors corresponding to the visual cue image, i is the index of the feature vector, and v i Let be the i-th eigenvector.
[0029] In the above scheme, the feature map contains multiple feature map patches; the step of using the aggregated feature vector to perform context matching on the feature map to obtain an enhanced image of the feature map includes:
[0030] For each feature map patch, a cross-attention mechanism is used to determine the similarity score between each feature map patch and the aggregated feature vector;
[0031] The similarity scores are normalized using a normalization function to obtain the weight corresponding to each feature map block;
[0032] The enhanced features of each feature map are determined based on the weights corresponding to each feature map patch and the aggregated feature vector, and the enhanced image of the feature map is determined using the enhanced features of each feature map patch.
[0033] In the above scheme, the cross-validation of the m review annotation results for each augmented image includes:
[0034] Extract all annotation boxes and their corresponding annotation types from the m audit annotation results;
[0035] All annotation box combinations are determined based on the annotation boxes in m review annotation results; each annotation box combination is formed by extracting any one annotation box from different review annotation results, and each annotation box combination contains two annotation boxes, which come from different review annotation results;
[0036] Determine the intersection-union ratio (CUIR) of each of the stated annotation box combinations, arrange all CUIRs in descending order, and determine the annotation box combinations with CUIRs greater than a preset CUIR threshold as the target annotation box combinations;
[0037] The global intersection-union ratio is determined based on the intersection-union ratio of each of the labeled box combinations. If the global intersection-union ratio is determined to be greater than a preset threshold, it is then determined whether the types of the two labeled boxes in the target labeled box combination are consistent.
[0038] If the two bounding boxes in all target bounding box combinations are of the same type, then the verification result is deemed to meet the requirements.
[0039] A second aspect of the present invention provides an image annotation apparatus, the apparatus comprising:
[0040] An expansion unit is used to expand the initial image dataset to obtain an expanded image dataset; the initial image dataset is a labeled dataset;
[0041] The annotation unit is used to pre-annotate the expanded image dataset based on the initial image dataset to obtain the pre-annotation result for each expanded image;
[0042] The review unit is used to review the pre-labeling results of each pre-labeled expanded image in the expanded image dataset, and obtain m review labeling results for each expanded image.
[0043] The verification unit is used to cross-verify the m review annotation results of each expanded image. If the verification results meet the requirements, the expanded image is determined to be successfully marked. The m is greater than or equal to 2 and is a positive integer.
[0044] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any of the first aspects.
[0045] A third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method described in any of the first aspects.
[0046] This invention provides an image annotation method, apparatus, medium, and device. The method includes: expanding an initial image dataset to obtain an expanded image dataset; the initial image dataset is an already labeled dataset; pre-annotating the expanded image dataset based on the initial image dataset to obtain a pre-annotation result for each expanded image; reviewing the pre-annotation results for each pre-annotated expanded image in the expanded image dataset to obtain m reviewed annotation results for each expanded image; cross-validating the m reviewed annotation results for each expanded image, and determining that the expanded image is successfully labeled if the validation results meet the requirements; where m is greater than or equal to 2 and is a positive integer. This allows for the collection of a small amount of initial image data, followed by expansion, thereby reducing the time cost of data collection. Furthermore, since the initial images in the initial image dataset are already labeled, the expanded images in the expanded image dataset can be automatically annotated based on the already labeled images. The entire annotation process only requires staff review, eliminating the need for extensive annotation time and thus improving annotation efficiency. Attached Figure Description
[0047] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0048] Figure 1 A schematic flowchart of an image annotation method according to an embodiment of the present invention is shown;
[0049] Figure 2 A schematic diagram of an image annotation device according to an embodiment of the present invention is shown. Detailed Implementation
[0050] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0051] This invention provides an image annotation method, such as... Figure 1 As shown, the method includes the following steps:
[0052] S110, the initial image dataset is expanded to obtain an expanded image dataset; the initial image dataset is a labeled dataset.
[0053] To reduce the time cost for staff to collect initial images and improve the efficiency of initial image data collection, this invention can collect a small number of initial images (e.g., 20 images) to form an initial image dataset. Based on pre-created annotation tasks, the initial images are assigned to one or more annotators for manual annotation, serving as visual cues. This facilitates subsequent automatic annotation of expanded images in the expanded image dataset based on the already annotated initial images in the initial image dataset.
[0054] If a single target object A needs to be labeled, 20 initial images need to be collected, and each of the 20 initial images contains only the target object A.
[0055] If multiple target objects need to be labeled, a separate initial image needs to be collected for each type of target object.
[0056] For example, if it is necessary to annotate both target object A and target object B at the same time, for target object A, 20 initial images need to be collected, each containing only target object A, forming the initial image dataset corresponding to target object A; for target object B, 20 initial images also need to be collected, each containing only target object B, forming the initial image dataset corresponding to target object B.
[0057] During manual annotation, annotators can draw annotation boxes for target objects in each initial image and add corresponding labels. For example, for an image of a steel rolling mill, if the initial image contains equipment such as rolling mills and cooling roller conveyors, annotation boxes need to be drawn for the rolling mill and cooling roller conveyors respectively, and corresponding labels need to be added to the annotation boxes. For example, the label corresponding to the annotation box drawn for the rolling mill is "rolling mill".
[0058] After all initial images are labeled, in order to improve the labeling quality, the pre-labeled initial images need to be sent to the reviewer for review. If it is determined that there are no problems with the pre-labeling of the initial images, the review will be approved; if it is determined that there are errors in the pre-labeling, the initial image will be returned and the reason for the return will be sent at the same time. The labeler needs to correct the labeling of the initial image.
[0059] If an initial image is severely blurry, contains heavily occluded objects, has an undetermined category, or contains an unknown category, it can be pre-labeled as a difficult sample. This sample will be submitted to the reviewer for priority review. When the reviewer confirms it as a difficult sample, they will delete the initial image from the initial image dataset and re-collect an initial image.
[0060] In one implementation, the initial image dataset is expanded to obtain an expanded image dataset, including:
[0061] For each initial image in the initial image dataset, a target image is searched in the historical sample database; the target image is an image whose similarity to the initial image meets a preset similarity threshold.
[0062] The target image is identified as the augmented image.
[0063] In one implementation, the initial image dataset is expanded, including:
[0064] Convert the labeled target objects in the initial image into 3D models;
[0065] Within a virtual 3D space, based on the parameters of a virtual camera and lighting conditions, a rendering engine is used to render the 3D model of the target object to obtain a composite image.
[0066] The synthesized image is identified as the augmented image.
[0067] Specifically, there are two main methods for expanding the initial image dataset. One method is to find sample images with high similarity to the initial images from the historical sample database and use these sample images as expanded images to form an expanded image dataset.
[0068] In this approach, the historical sample library needs to be traversed, and each sample image in the historical sample library is matched with each initial image for similarity to obtain the similarity between the initial image and the sample image. If the similarity is greater than or equal to a preset similarity threshold (e.g., 80%), then the sample image is determined as the augmented image.
[0069] Another method to augment the initial image dataset is to use an arbitrary viewpoint image generation model. This augmentation method requires converting the labeled target objects in the initial images into virtual 3D models, and then importing the 3D models of the target objects into the virtual 3D space as objects for subsequent image generation.
[0070] In a virtual 3D space, the parameters and lighting conditions of the virtual camera are adjusted. The virtual camera is moved around the target object, and three key shooting parameters (angle, distance, and lighting) are adjusted. The image generation model is invoked, and its rendering engine is used to render and image the target object, resulting in a composite image. This composite image is then designated as the augmented image.
[0071] This image generation method reduces users' data collection workload by 66%.
[0072] S111, pre-label the expanded image dataset according to the initial image dataset to obtain the pre-labeling result of each expanded image.
[0073] Once the expanded image dataset is determined, the expanded images in the expanded image dataset need to be labeled. To improve labeling efficiency, this invention can pre-label the expanded image dataset based on the initial image dataset, obtaining the pre-labeling results for each expanded image.
[0074] It should be noted that if the augmented image contains only a single target object, only the single target object needs to be pre-annotated. The augmented image dataset can be pre-annotated based on the initial image dataset containing the single target object.
[0075] If the augmented image contains multiple objects (e.g., object A and object B), object A and object B need to be pre-labeled separately. This can be done sequentially using the initial image dataset containing object A and the initial image dataset containing object B.
[0076] For example, target object A in the augmented image can be pre-labeled based on the initial image dataset containing target object A. After the pre-labeling of target object A in the augmented image is completed, target object B in the augmented image can be pre-labeled based on the initial image dataset containing target object B.
[0077] For ease of description, the following explanation uses the pre-annotation method for single-target objects (for multi-target objects, the pre-annotation method is the same for each pre-annotation as for single-target objects).
[0078] In one implementation, the augmented image dataset is labeled based on the initial image dataset, including:
[0079] Obtain the initial images from the initial image dataset and the first mask image for each initial image;
[0080] The aggregated feature vector of all initial images is determined based on each initial image and the first mask image of each initial image;
[0081] Each augmented image in the augmented image dataset is converted into a corresponding feature map;
[0082] For each combination of aggregated feature vector and feature map, the aggregated feature vector in the combination is used to perform context matching on the feature map to obtain an enhanced image of the feature map;
[0083] The enhanced image is decoded to obtain the second mask image corresponding to the enhanced image and the probability of each region in the second mask image. Regions with a probability greater than a preset probability threshold are determined as target regions.
[0084] The target region is labeled with its type based on the label type in the aggregated feature vector, and a label box is drawn for the target region.
[0085] In one implementation, determining an aggregated feature vector based on each initial image and a first mask image of each initial image includes:
[0086] For each initial image, the first mask image of the initial image is superimposed on the initial image to obtain the visual cue image;
[0087] Each visual cue image is encoded using a pre-trained visual cue encoder to obtain the feature vector corresponding to each visual cue image;
[0088] According to the formula The feature vectors corresponding to each visual cue image are aggregated to obtain the aggregated feature vector P. store ;in,
[0089] n is the total number of feature vectors corresponding to the visual cue image, i is the index of the feature vector, and v i Let be the i-th eigenvector.
[0090] In one implementation, the feature map comprises multiple feature map patches; context matching is performed on the feature map using aggregated feature vectors to obtain an enhanced image of the feature map, including:
[0091] For each feature map patch, a cross-attention mechanism is used to determine the similarity score between each feature map patch and the aggregated feature vector;
[0092] The similarity scores are normalized using a normalization function to obtain the weight corresponding to each feature map patch;
[0093] The enhancement features of each feature map patch are determined based on the weights and aggregated feature vectors corresponding to each feature map patch, and the enhancement image of the feature map is determined using the enhancement features of each feature map patch.
[0094] Specifically, when manually annotating each initial image in the initial image dataset, it is also necessary to outline the target object, ensuring complete coverage of the target object and excluding the background. The outlined area is marked as the foreground (target), and the remaining area is marked as the background, generating a corresponding first mask image (typically, the foreground pixel value is 255, and the background is 0). The target object type is consistent in all initial images.
[0095] Then, the first mask image is overlaid on the corresponding initial image to obtain the visual cue image (highlight image), which highlights the target object in the visual cue image. Each highlight image is encoded using a pre-trained visual cue encoder (a visual cue encoder based on a mask autoencoder), obtaining the feature vector corresponding to the target object in each highlight image. This feature vector is then used with the formula... Aggregate the n feature vectors to obtain the aggregated feature vector.
[0096] When each augmented image in the augmented image dataset needs to be labeled, the augmented image is uniformly divided into multiple non-overlapping patches. A pre-trained visual cue encoder is used to encode these non-overlapping patches to obtain corresponding feature vectors. All feature vectors are then arranged according to the spatial order of the patches in the original augmented image to form a feature map F. new .
[0097] Since there are many feature maps and aggregated feature vectors, to avoid missing context matching, it is necessary to traverse multiple feature maps and multiple aggregated feature vectors, combining any feature map and any aggregated feature vector to obtain multiple combinations (combinations of aggregated feature vectors and feature maps). For each combination, context matching is performed on the feature map using the aggregated feature vector in the combination to obtain the enhanced image of the feature map.
[0098] When performing in-context matching, since the feature map contains multiple feature map patches, a cross-attention mechanism is needed to determine the similarity score between each feature map patch and the aggregated feature vector, as follows:
[0099] The cross-attention mechanism involves three roles: Q (Query), K (Key), and V (Value), where Q is the feature map F. new For each feature vector in the vector, K and V are aggregated feature vectors of type P. store ;
[0100] Using the dot product formula QK T Determine feature map F new The similarity score between each feature vector and the aggregate feature vector is considered; the larger the dot product result, the better the feature map F. new The more similar the feature vector Q and the aggregate feature vector K are, the better.
[0101] To avoid the dot product of high-dimensional vectors becoming too large and to prevent the subsequent Softmax normalization function from outputting extreme values (either all 1s or all 0s), the dot product needs to be divided by the square root of the dimension of the feature vector Q. That is, using... Determine feature map F new The similarity score between each feature vector and the aggregate feature vector.
[0102] Using normalization function The similarity scores are normalized to obtain the weight (between 0 and 1) for each feature patch. The sum of the weights of all feature patches is 1. The feature patch with the higher similarity score has a larger weight, and the feature patch with the lower similarity score has a smaller weight.
[0103] Then, the weights corresponding to each feature patch are multiplied by the aggregated feature vector to obtain the enhanced features of the corresponding feature patch. Because V is a common feature, feature patches with high weights are strongly empowered by V, making the feature patches closer to V; feature patches with low weights (such as the background) receive almost no empowerment from V, and their features are suppressed. After the enhanced features of each feature patch are determined, it is equivalent to obtaining an enhanced image of the feature map. In the final output feature map, the features of the target object region are significantly enhanced, while the features of the background region are weakened.
[0104] The enhanced image is decoded using a decoder, and after upsampling and convolution, a second mask image with the same size as the original image is output, along with the probability of each region in the second mask image. When the probability of a certain region is greater than a preset probability threshold (e.g., 0.5), it indicates that a target object has been detected. At this point, the type of the target region can be labeled according to the label type in the aggregated feature vector, and the type of the target region is consistent with the type of the target object in the aggregated feature vector.
[0105] Simultaneously, bounding boxes can be drawn for the target area, as follows: the pixel value of the target area is marked as 255, and the pixel value of non-target areas is marked as 0, thus highlighting the outline of the target area. An edge detection algorithm is used to detect the outline of the target area, and corresponding bounding boxes are generated along the detected outline.
[0106] This allows for rapid annotation of each augmented image in the augmented image dataset, improving annotation efficiency. By augmenting the initial image dataset using the above method, well-annotated augmented image samples can be generated quickly, reducing the annotation workload by 33%.
[0107] S112, for each pre-annotated expanded image in the expanded image dataset, the pre-annotation results of the expanded image are reviewed to obtain m reviewed annotation results for each expanded image; the m reviewed annotation results of each expanded image are cross-validated, and if the verification results meet the requirements, the expanded image is determined to be successfully labeled.
[0108] To improve annotation quality, for each pre-annotated expanded image, m reviewers are assigned to review the pre-annotation results, resulting in m reviewed annotation results for each expanded image. m is greater than or equal to 2 and is a positive integer.
[0109] Furthermore, the quality of manual review is difficult to guarantee. Different reviewers may have differing interpretations of the same image, leading to inconsistent review results. Therefore, it is necessary to cross-validate the m review annotation results for each expanded image. If the validation results meet the requirements, the expanded image is considered successfully labeled.
[0110] In one implementation, cross-validation is performed on the m review annotation results for each augmented image, including:
[0111] Extract all annotation boxes and their corresponding annotation types from m audit annotation results;
[0112] All annotation box combinations are determined based on the annotation boxes in m audit annotation results; each annotation box combination is formed by extracting one annotation box from each different audit annotation result, and each annotation box combination contains two annotation boxes, which come from different audit annotation results;
[0113] Determine the intersection-union ratio (CUIR) of each bounding box combination, arrange all CUIRs in descending order, and determine the bounding box combinations with CUIRs greater than a preset CUIR threshold as the target bounding box combinations;
[0114] The global intersection-union ratio is determined based on the intersection-union ratio of each group of annotation boxes. If the global intersection-union ratio is determined to be greater than a preset threshold, it is then determined whether the types of the two annotation boxes in the target annotation box group are consistent.
[0115] If the two bounding boxes in all target bounding box combinations are of the same type, then the verification result is considered to meet the requirements.
[0116] Specifically, the Intersection over Union (IoU) ratio is a core quantitative indicator for measuring the degree of overlap between two regions. Therefore, the IoU ratio is a key basis for determining whether the positions of two bounding boxes are consistent.
[0117] For m review annotation results of the same expanded image, it is necessary to first ensure that the boundary positions of the n images are consistent, and then extract all the annotation boxes and corresponding annotation types from the m review annotation results respectively.
[0118] For example, taking m=2 as an example, if it is a multi-target object annotation, the first review result contains multiple annotation boxes, and the second review result also contains multiple annotation boxes. The annotation boxes in the first review annotation result can be extracted to form annotation box set A; all annotation boxes in the second review result can be extracted to form annotation box set B.
[0119] For a combination of annotation frames formed by annotation frames in annotation frame set A and annotation frame set B (a combination formed by an annotation frame from annotation frame set A and an annotation frame from annotation frame set B), determine the intersection area S1 of the overlapping areas of the two annotation frames in the combination (if there is no overlap, the intersection area is 0), and determine the union area S2 of the two annotation frames (the union area is the difference between the sum of the areas of the two annotation frames and the intersection area); according to the formula Determine the intersection-union ratio (IoU) of the two annotation boxes in the annotation box combination.
[0120] Following the same method described above, the intersection-union ratio (CUR) of all bounding box combinations can be determined. Combinations with CURs greater than a preset threshold (e.g., 0.5) are then identified as target bounding box combinations. The average CUR of all target bounding box combinations is taken as the global CUR. If the global CUR is greater than the preset threshold (e.g., 0.5), it is necessary to further determine whether the types of the two bounding boxes in the target bounding box combinations are consistent. If the types of the two bounding boxes in all target bounding box combinations are consistent, then the verification result meets the requirements.
[0121] Additionally, if the global intersection-union ratio (IU) is determined to be less than or equal to the preset IU threshold, it indicates that the verification result does not meet the requirements. The expanded image is then marked as pending review and sent to the review supervisor for further review. If the review supervisor determines that the pre-annotation results of the expanded image are accurate, the review status is updated to "approved." If the review supervisor determines that the pre-annotation results of the expanded image are inaccurate, the expanded image is returned to the annotator for correction.
[0122] If the annotation is for a single target object, the first review result contains only one annotation box, and the second review result also contains only one annotation box. In this case, the annotation box combination contains only one set. If it is determined that the intersection-union ratio of the annotation box combination is greater than the preset intersection-union ratio threshold (e.g., 0.5), and the two annotation boxes are of the same type, then the verification result is determined to meet the requirements.
[0123] This ensures that the accuracy of the annotation reaches over 99% and reduces the workload of the annotator by 90%.
[0124] Based on the same inventive concept as in the foregoing embodiments, this embodiment also provides an image annotation device, such as... Figure 2 As shown, the device includes:
[0125] The expansion unit 21 is used to expand the initial image dataset to obtain an expanded image dataset; the initial image dataset is a labeled dataset;
[0126] The annotation unit 22 is used to pre-annotate the expanded image dataset according to the initial image dataset to obtain the pre-annotation result of each expanded image;
[0127] The review unit 23 is used to review the pre-labeling results of each pre-labeled expanded image in the expanded image dataset, and obtain m review labeling results for each expanded image.
[0128] Verification unit 24 is used to cross-verify the m review annotation results of each expanded image. If the verification results meet the requirements, the expanded image is determined to be successfully marked. The n is greater than or equal to 2 and is a positive integer.
[0129] Since the apparatus described in the embodiments of this invention is an apparatus used to implement the image annotation method of the embodiments of this invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in the embodiments of this invention, and therefore will not be described in detail here. All apparatuses used in the methods of the embodiments of this invention fall within the scope of protection of this invention.
[0130] Based on the same inventive concept, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any step of the method described above.
[0131] Based on the same inventive concept, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.
[0132] Through one or more embodiments of the present invention, the present invention has the following beneficial effects or advantages:
[0133] This invention provides an image annotation method, apparatus, medium, and device. The method includes: expanding an initial image dataset to obtain an expanded image dataset; the initial image dataset is an already labeled dataset; pre-annotating the expanded image dataset based on the initial image dataset to obtain a pre-annotation result for each expanded image; reviewing the pre-annotation results for each pre-annotated expanded image in the expanded image dataset to obtain m reviewed annotation results for each expanded image; cross-validating the m reviewed annotation results for each expanded image, and determining that the expanded image is successfully labeled if the verification results meet the requirements; where m is greater than or equal to 2 and is a positive integer. This allows for the collection of a small amount of initial image data, followed by expansion, thereby reducing the time cost of data collection. Furthermore, since the initial images in the initial image dataset are already labeled, the expanded images in the expanded image dataset can be automatically annotated, reviewed, and verified based on the already labeled images. The entire annotation process only requires staff review, eliminating the need for extensive annotation time and thus improving annotation efficiency.
[0134] In addition, this invention introduces a cross-validation mechanism to automatically discover and filter out controversial annotations, reducing the workload of reviewers by more than 90% while ensuring that the accuracy of the final dataset reaches more than 99%.
[0135] Image augmentation technology can be used to expand the sample library by more than 2 times at low cost, effectively solving the problem of difficult data collection in specific scenarios.
[0136] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages is for the purpose of disclosing the best mode of implementation of the invention.
[0137] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0138] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.
[0139] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.
[0140] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.
[0141] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components of the gateway, proxy server, or system according to embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.
[0142] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
[0143] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An image labeling method, characterized by, The method includes: The initial image dataset is expanded to obtain an expanded image dataset; the initial image dataset is a labeled dataset. The expanded image dataset is pre-annotated based on the initial image dataset to obtain the pre-annotation result for each expanded image; For each pre-annotated augmented image in the augmented image dataset, the pre-annotation results of the augmented image are reviewed to obtain m reviewed annotation results for each augmented image; the m reviewed annotation results of each augmented image are cross-validated, and if the verification results meet the requirements, the augmented image is determined to be successfully labeled; m is greater than or equal to 2 and is a positive integer.
2. The method of claim 1, wherein, The expansion of the initial image dataset includes: Based on each initial image in the initial image dataset, a target image is searched in the historical sample database; the target image is an image whose similarity to the initial image meets a preset similarity threshold. The target image is identified as an augmented image.
3. The method of claim 1, wherein, The expansion of the initial image dataset includes: Convert the labeled target objects in the initial image into three-dimensional models; Within a virtual 3D space, based on the parameters of the virtual camera and the lighting conditions, a rendering engine is used to render the 3D model of the target object to obtain a composite image. The synthesized image is identified as an augmented image.
4. The method of claim 1, wherein, The step of annotating the expanded image dataset based on the initial image dataset includes: Obtain each initial image from the initial image dataset and the first mask image of each initial image; The aggregated feature vector of all initial images is determined based on each initial image and the first mask image of each initial image; Each augmented image in the augmented image dataset is converted into a corresponding feature map; For each combination of aggregated feature vector and feature map, the aggregated feature vector in the combination is used to perform context matching on the feature map to obtain an enhanced image of the feature map; The enhanced image is decoded to obtain the second mask image corresponding to the enhanced image and the probability of each region in the second mask image. Regions with a probability greater than a preset probability threshold are determined as target regions. The target region is labeled with its type based on the label type in the aggregated feature vector, and a label box is drawn for the target region.
5. The method of claim 4, wherein, The step of determining the aggregated feature vector based on each initial image and the first mask image of each initial image includes: For each initial image, the first mask image of the initial image is superimposed on the initial image to obtain a visual cue image; Each visual cue image is encoded using a pre-trained visual cue encoder to obtain the feature vector corresponding to each visual cue image; According to the formula The feature vectors corresponding to each of the visual cue images are aggregated to obtain an aggregated feature vector P store ; wherein, n is the total number of feature vectors corresponding to the visual cue image, i is the serial number of the feature vector, v i is the ith feature vector.
6. The method of claim 4, wherein, The feature map contains multiple feature map patches; the step of performing context matching on the feature map using the aggregated feature vector to obtain an enhanced image of the feature map includes: For each feature map patch, a cross-attention mechanism is used to determine the similarity score between each feature map patch and the aggregated feature vector; The similarity scores are normalized using a normalization function to obtain the weight corresponding to each feature map block; The enhanced features of each feature map are determined based on the weights corresponding to each feature map patch and the aggregated feature vector, and the enhanced image of the feature map is determined using the enhanced features of each feature map patch.
7. The method as described in claim 1, characterized in that, The cross-validation of the m review annotation results for each augmented image includes: Extract all annotation boxes and their corresponding annotation types from the m audit annotation results; All annotation box combinations are determined based on the annotation boxes in m audit annotation results; each annotation box combination is formed by extracting any one annotation box from different audit annotation results, and each annotation box combination contains two annotation boxes, which come from different audit annotation results. Determine the intersection-union ratio (CUIR) of each of the stated annotation box combinations, arrange all CUIRs in descending order, and determine the annotation box combinations with CUIRs greater than a preset CUIR threshold as the target annotation box combinations; The global intersection-union ratio is determined based on the intersection-union ratio of each of the labeled box combinations. If the global intersection-union ratio is determined to be greater than a preset threshold, it is then determined whether the types of the two labeled boxes in the target labeled box combination are consistent. If the two bounding boxes in all target bounding box combinations are of the same type, then the verification result is deemed to meet the requirements.
8. An image annotation device, characterized in that, The device includes: An expansion unit is used to expand the initial image dataset to obtain an expanded image dataset; the initial image dataset is a labeled dataset; The annotation unit is used to pre-annotate the expanded image dataset based on the initial image dataset to obtain the pre-annotation result for each expanded image; The review unit is used to review the pre-labeling results of each pre-labeled expanded image in the expanded image dataset, and obtain m review labeling results for each expanded image. The verification unit is used to cross-verify the m review annotation results of each expanded image. If the verification results meet the requirements, the expanded image is determined to be successfully marked. The m is greater than or equal to 2 and is a positive integer.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-7.
10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-7.