Image labeling method and device, equipment and storage medium
By using a self-supervised large model in image annotation for feature extraction and combining target downstream task headers for annotation, the problem of low image annotation efficiency in the prior art is solved, and a more efficient automatic annotation process is achieved.
Patent Information
- Application Number
- CN202311344625.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-17
- Publication Date
- 2025-04-18
AI Technical Summary
In the prior art, image labeling efficiency is low, mainly due to the low manual labeling efficiency, resulting in low efficiency in training and labeling processes for specific tasks.
The self-supervised large model is used to extract image features, and the target downstream task head is used to label image features, simplify the labeling process and improve the labeling efficiency.
By combining self-supervised large models and target downstream task heads, the overall labeling efficiency of automatic labeling is improved, model training time is saved, and the annotation process is simplified.
Smart Images
Figure CN119851273A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data annotation technology, and in particular, relates to an image annotation method, device, equipment and storage medium. Background Art
[0002] Image annotation is the process of labeling images in a dataset to train a machine learning model. Image annotation can include annotation types such as image classification, object detection, object recognition, image segmentation, etc. For each annotation type, labeling different annotation objects can belong to different annotation tasks.
[0003] At present, image annotation is usually performed through a specific task model. Specifically, for different annotation tasks, it is necessary to first obtain a large amount of manually annotated image data to train the initial model and obtain a specific task model. Then, the specific task model is used for annotation. Due to the low efficiency of manual annotation, the overall annotation efficiency is low when using a specific task model for automatic annotation. Summary of the invention
[0004] The embodiments of the present application provide an image annotation method, apparatus, device and storage medium, which can improve the overall annotation efficiency of automatic annotation.
[0005] In a first aspect, an embodiment of the present application provides an image annotation method, the method comprising:
[0006] Acquire a first image to be annotated;
[0007] Extracting features from the first image using a self-supervised large model to obtain image features of the first image;
[0008] The target downstream task head is used to perform image annotation corresponding to a target annotation task on the image features to obtain an annotation result of the first image, wherein the target annotation task is any one of a plurality of computer vision annotation tasks, and the target downstream task head is a neural network structure for processing the target annotation task.
[0009] In a possible implementation, the target labeling task includes an image classification task, the labeling result includes a category label corresponding to the first image, and after obtaining the labeling result of the first image, the method further includes:
[0010] Obtaining a first confidence level corresponding to the category label;
[0011] When the first confidence level is greater than a first preset threshold, the target downstream task head is iteratively optimized using the first image and its corresponding category label.
[0012] In a possible implementation, the target labeling task includes a target detection task, the labeling result includes a category and a detection box of each object in the first image, and after obtaining the labeling result of the first image, the method further includes:
[0013] Obtaining a second confidence level corresponding to each of the detection frames;
[0014] When the second confidence level is greater than a second preset threshold, cropping the first image according to the detection frame to obtain at least one second image;
[0015] For each of the second images, use a large image-text matching model to perform image-text matching on the second image and the category text of the second image to obtain a first matching degree;
[0016] When the first matching degree is greater than a third preset threshold, the target downstream task head is iteratively optimized using the second image and its corresponding category.
[0017] In a possible implementation, the target labeling task includes an image segmentation task, the labeling result includes a category and a pixel of each object in the first image, and after obtaining the labeling result of the first image, the method further includes:
[0018] Obtaining a third confidence level corresponding to each of the pixels;
[0019] When the third confidence level is greater than a fourth preset threshold, cropping the first image according to the pixels to obtain a plurality of third images;
[0020] For each of the third images, use a large image-text matching model to perform image-text matching on the third image and the category text of the third image to obtain a second matching degree;
[0021] When the second matching degree is greater than a fifth preset threshold, the target downstream task head is iteratively optimized using the third image and its corresponding category.
[0022] In a possible implementation, the annotation result is a segmentation heat map, the segmentation heat map includes a plurality of pixels, and the step of cropping the first image according to the pixels includes:
[0023] Segment the segmentation heat map according to the multiple pixels to obtain multiple regions;
[0024] mapping the plurality of regions into the first image;
[0025] The first image is cropped according to the multiple regions.
[0026] In a possible implementation manner, segmenting the segmentation heat map according to the multiple pixels includes:
[0027] Determine the minimum bounding rectangle corresponding to each of the plurality of pixels;
[0028] The segmentation heat map is segmented according to a plurality of the minimum circumscribed rectangles.
[0029] In a possible implementation manner, after obtaining the annotation result of the first image, the method further includes:
[0030] Acquire a training data set according to a preset period, wherein the training data set includes images of multiple vehicle cockpits;
[0031] The self-supervised large model is iteratively optimized using the training data set.
[0032] In a second aspect, an embodiment of the present application provides an image annotation device, the device comprising:
[0033] A first acquisition module, used to acquire a first image to be labeled;
[0034] An extraction module, used for performing feature extraction on the first image using a self-supervised large model to obtain image features of the first image;
[0035] A labeling module is used to use a target downstream task head to perform image labeling corresponding to a target labeling task on the image features to obtain a labeling result of the first image, wherein the target labeling task is any one of a plurality of computer vision labeling tasks, and the target downstream task head is a neural network structure for processing the target labeling task.
[0036] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;
[0037] When the processor executes the computer program instructions, the processor implements any possible implementation method of the first aspect described above.
[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements a method in any possible implementation method of the first aspect described above.
[0039] In the image annotation method, device, equipment and storage medium of the embodiment of the present application, since the self-supervised large model is a large model with strong generalization ability, there is no need to perform additional training on the self-supervised large model, and the self-supervised large model can be used to extract features of the first image to obtain the image features of the first image. Since there is no need to perform additional training on the large model, it is possible to save time for model training and improve annotation efficiency. By using the target downstream task head to annotate the image features instead of the first image with the image corresponding to the target annotation task, the annotation process of the target downstream task head can be simplified and the annotation efficiency of the target downstream task head can be improved. In this way, through the embodiment of the present application, the overall annotation efficiency of automatic annotation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0041] Figure 1 It is a flowchart of an image annotation method provided in an embodiment of the present application;
[0042] Figure 2 is a schematic diagram of an image annotation method provided in an embodiment of the present application;
[0043] Figure 3 is a structural schematic diagram of an image annotation device provided in an embodiment of the present application;
[0044] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0046] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.
[0047] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0048] As described in the background technology section, in order to solve the problems of the prior art, the embodiments of the present application provide an image annotation method, device, equipment and storage medium. The image annotation method can be applied to the scene of annotating images in a vehicle cabin.
[0049] The following first introduces the image annotation method provided in the embodiment of the present application.
[0050] Figure 1 A flow chart of an image annotation method provided by an embodiment of the present application is shown. The image annotation method can be performed by an automatic annotation model. The automatic annotation model may include a first neural network structure and a second neural network structure. The first neural network structure may be used as a self-supervised large model for image feature extraction. The second neural network structure may be used as a downstream task head for performing a specific image annotation task.
[0051] like Figure 1 As shown, the image annotation method provided in the embodiment of the present application includes the following steps:
[0052] S110, obtaining a first image to be labeled;
[0053] S120, extracting features from the first image using the self-supervised large model to obtain image features of the first image;
[0054] S130. Use the target downstream task head to perform image annotation corresponding to the target annotation task on the image features to obtain the annotation result of the first image, where the target annotation task is any one of a plurality of computer vision annotation tasks, and the target downstream task head is a neural network structure for processing the target annotation task.
[0055] In the image annotation method of the embodiment of the present application, since the self-supervised large model is a large model with strong generalization ability, there is no need to perform additional training on the self-supervised large model, and the self-supervised large model can be used to extract features of the first image to obtain the image features of the first image. Since there is no need to perform additional training on the large model, it is possible to save time for model training and improve annotation efficiency. By using the target downstream task head to annotate the image features instead of the first image with the image corresponding to the target annotation task, the annotation process of the target downstream task head can be simplified and the annotation efficiency of the target downstream task head can be improved. In this way, through the embodiment of the present application, the overall annotation efficiency of automatic annotation can be improved.
[0056] The specific implementation methods of the above steps are introduced below.
[0057] In some embodiments, in S110, the first image may be an image in a vehicle cabin. The vehicle cabin may be a cabin section inside the vehicle for users to sit in. In addition, the first image may be a color image or a black and white image, which is not limited here.
[0058] As an example, when the camera in the vehicle cabin is an infrared camera, an infrared image captured by the infrared camera can be first acquired, and then the infrared image can be converted into a grayscale image, and the grayscale image can be used as the first image.
[0059] As another example, when the camera in the vehicle cabin is a color camera, a color image captured by the color camera can be directly used as the first image.
[0060] In some embodiments, in S120, relative to ordinary deep learning models, the parameter scale and training data scale of the large model are both larger. Since the parameter scale and training data scale of the large model are both large, the large model usually has a strong generalization ability. The self-supervised large model can be a large model that can be trained by the image itself without the need for labeled data. Among them, the training process can be a process that enables the model to learn how to extract image features. Since the training process of the self-supervised large model does not require labeled data, the process of training the self-supervised large model saves a lot of labeling costs. In addition, the self-supervised large model can be implemented based on a vision transformer (Vision Transformer, VIT). VIT can be a neural network structure including a normalization layer, a multi-head attention layer and a multi-layer perception layer.
[0061] As an example, after the first image is input into the self-supervised large model, the self-supervised large model may extract features of the first image through VIT to obtain image features of the first image.
[0062] As an example, if the first image is a grayscale image, before the first image is input into the self-supervised large model, a large number of grayscale images can be obtained, and then the self-supervised large model can be trained using the multiple grayscale images. The multiple grayscale images can include images obtained by grayscale conversion of infrared images in the vehicle cabin, and images obtained by grayscale conversion of open source color images. In addition, the specific training process can refer to the existing method, which will not be described in detail here.
[0063] Based on this, in order to continuously improve the accuracy of feature extraction by the self-supervised large model, and thus continuously improve the accuracy of image annotation, in some embodiments, the following may also be included:
[0064] Acquire a training data set according to a preset period, wherein the training data set includes images of multiple vehicle cockpits;
[0065] Iteratively optimize the self-supervised large model using the training dataset.
[0066] Here, the preset period may be, for example, one day, one week, two weeks, etc. At every preset period, multiple images in the vehicle cockpit may be collected once, and the multiple images may be used as a training data set. If the first image is a color image, the multiple images in the vehicle cockpit may be color images. If the first image is a grayscale image, the multiple images in the vehicle cockpit may be grayscale images.
[0067] In this way, by iteratively optimizing the self-supervised large model every preset period, the accuracy of feature extraction of the self-supervised large model can be continuously improved, thereby continuously improving the accuracy of image annotation.
[0068] In some embodiments, in S130, the plurality of computer vision annotations may include all vision-related annotation tasks such as image classification, target detection, image segmentation, depth estimation, etc. The target annotation task may be a specific downstream task. The downstream task may be any one of the plurality of computer vision annotation tasks.
[0069] In addition, the downstream task head can be a neural network structure for processing downstream tasks. The neural network structure can specifically be a stack of several convolutional layers. The target downstream task head can be a neural network structure for processing target labeling tasks. In order for the target downstream task head to process the target labeling task, the initial neural network structure can be fine-tuned (trained) in advance using the target labeling data to obtain the target downstream task head. Among them, the target labeling data can be the labeling data corresponding to the target labeling task. The target labeling data can be obtained by manual labeling. Since the input of the downstream task head is the image features output by the self-supervised large model, and the self-supervised large model is obtained through a large number of image training, therefore, for any downstream scenario, only a small amount of target labeling data is needed to fine-tune the target downstream task head to obtain an automatic labeling model for the target labeling task.
[0070] As an example, if the target labeling task is an image classification task, the labeling data may include the image and its corresponding category label. If the target labeling task is an object detection task, the labeling data may include the image, and the category and detection box corresponding to each object in the image. If the target labeling task is an image segmentation task, the labeling data may include the image, and the category and pixel corresponding to each object in the image.
[0071] As an example, after outputting the image features, the self-supervised large model can input the image features into the target downstream task head. By using the target downstream task head to annotate the image features corresponding to the target annotation task, the annotation result of the first image can be obtained. Specifically, when the target annotation task is an image classification task, the annotation result can be a category label corresponding to the first image. When the target annotation task is a target detection task, the annotation result can include the category and detection box of each object in the first image. When the target annotation task is an image segmentation task, the annotation result can include the category and pixel of each object in the first image.
[0072] As an example, when the target downstream task head outputs the labeling result, it can also output the confidence corresponding to the labeling result. Among them, the confidence can be used to evaluate the accuracy of the labeling result. When the confidence is greater than the preset confidence threshold, the labeling result can be determined as an accurate labeling result and stored. When the confidence is less than or equal to the preset confidence threshold, the labeling result can be considered inaccurate and discarded.
[0073] Based on this, when the target labeling task is an image classification task, in order to continuously improve the labeling performance of the target downstream task head and continuously improve the accuracy of the labeling results, in some embodiments, after the above S130, the following may also be included:
[0074] Obtain a first confidence level corresponding to the category label;
[0075] When the first confidence level is greater than a first preset threshold, the target downstream task head is iteratively optimized using the first image and its corresponding category label.
[0076] Here, while the target downstream task head outputs the category label, it can also output a first confidence level corresponding to the category label. The first confidence level can be used to evaluate the accuracy of the category label. In addition, the first preset threshold value can be greater than or equal to the preset confidence level threshold value.
[0077] In this way, when the first confidence level is greater than the first preset threshold, the accuracy of the category label can be guaranteed. By iteratively optimizing the target downstream task head using the first image and its corresponding category label, the labeling performance of the target downstream task head can be continuously improved, and the accuracy of the labeling result can be continuously improved.
[0078] Based on this, when the target labeling task is a target detection task, in order to continuously improve the labeling performance of the target downstream task head and continuously improve the accuracy of the labeling results, in some embodiments, after the above S130, the following may also be included:
[0079] Get the second confidence level corresponding to each detection box;
[0080] When the second confidence level is greater than a second preset threshold, cropping the first image according to the detection frame to obtain at least one second image;
[0081] For each second image, use the large image-text matching model to perform image-text matching on the second image and the category text of the second image to obtain a first matching degree;
[0082] When the first matching degree is greater than a third preset threshold, the target downstream task head is iteratively optimized using the second image and its corresponding category.
[0083] Here, while the target downstream task head outputs the category and detection frame of each object in the first image, it can also output a second confidence level corresponding to the detection frame. The second confidence level can be used to evaluate the accuracy of the detection frame. In addition, the second preset threshold value can be greater than or equal to the preset confidence threshold value. The second preset threshold value can be the same as or different from the first preset threshold value, which is not limited here.
[0084] In addition, the target detection task can be implemented based on the target detection algorithm. The target detection algorithm can detect objects such as people, animals, plants, and objects in the image, and obtain the category of the object and the location of the object in the image. Among them, the location of the object in the image can be displayed in the form of a detection box.
[0085] Since the second image is obtained by cropping the first image according to the detection frame, and each detection frame may correspond to an object, each second image may include an object. The second image may include the entire object or a part of the object, which is not limited here.
[0086] In addition, the image-text matching large model can be a large model implemented based on an image-text matching algorithm. The image-text matching algorithm can match images and texts to obtain the similarity between the two. Among them, the text can be a category text. That is, if the image matches the category text, it can be determined that the image belongs to the category corresponding to the category text.
[0087] As an example, the third preset threshold may be a preset matching threshold between the image and the text. The third preset threshold may be, for example, 95%. If the first matching degree between the second image and the category text of the second image is greater than the third preset threshold, it can be determined that the second image and the category text of the second image are matched, and then it can be determined that the target downstream task head's annotation of the second image is accurate, and then the target downstream task head can be iteratively optimized using the second image and its corresponding category to continuously improve the annotation performance of the target downstream task head and continuously improve the accuracy of the annotation results.
[0088] In addition, the second image and its corresponding category can be used to iteratively optimize the large image-text matching model to continuously improve the matching accuracy of the large image-text matching model, thereby continuously improving the accuracy of the annotation results.
[0089] Based on this, when the target labeling task is an image segmentation task, in order to continuously improve the labeling performance of the target downstream task head and continuously improve the accuracy of the labeling results, in some embodiments, after the above S130, the following may also be included:
[0090] Obtain the third confidence level corresponding to each pixel;
[0091] When the third confidence level is greater than a fourth preset threshold, cropping the first image according to pixels to obtain a plurality of third images;
[0092] For each third image, use the image-text matching large model to perform image-text matching on the third image and the category text of the third image to obtain a second matching degree;
[0093] When the second matching degree is greater than a fifth preset threshold, the target downstream task head is iteratively optimized using the third image and its corresponding category.
[0094] Here, while the target downstream task head outputs the category and pixel of each object in the first image, it can also output a third confidence level corresponding to the pixel. The third confidence level can be used to evaluate the accuracy of the pixel. In addition, the fourth preset threshold value can be greater than or equal to the preset confidence level threshold value. The fourth preset threshold value can be the same as or different from the first preset threshold value and / or the second preset threshold value, which is not limited here.
[0095] In addition, the image segmentation task can be implemented by an image segmentation algorithm. The image segmentation algorithm can be used to locate objects and boundaries (lines, curves, etc.) in the image, divide an image into multiple parts, and obtain a segmentation heat map. The segmentation heat map may include the category corresponding to each object. Specifically, the image segmentation algorithm can be performed at the pixel level, assigning each pixel in the image to a specific object. Multiple pixels can form a pixel corresponding to the specific object. That is, each pixel can correspond to a specific object.
[0096] Since the third image is obtained by cropping the first image according to pixels, and each pixel may correspond to an object. Therefore, each third image may include an object. The third image may include the entire object or a part of the object, which is not limited here. The object may be any one of a person, an animal, and a plant.
[0097] In addition, the image-text matching large model can be a large model implemented based on an image-text matching algorithm. The image-text matching algorithm can match images and texts to obtain the similarity between the two. Among them, the text can be a category text. That is, if the image matches the category text, it can be determined that the image belongs to the category corresponding to the category text.
[0098] As an example, the fifth preset threshold may be a preset matching threshold between the image and the text. The fifth preset threshold may be, for example, 95%. If the second matching degree between the third image and the category text of the third image is greater than the fifth preset threshold, it can be determined that the third image and the category text of the third image are matched, and then it can be determined that the target downstream task head's annotation of the third image is accurate, and then the target downstream task head can be iteratively optimized using the third image and its corresponding category to continuously improve the annotation performance of the target downstream task head and continuously improve the accuracy of the annotation results.
[0099] In addition, the third image and its corresponding category can be used to iteratively optimize the large image-text matching model to continuously improve the matching accuracy of the large image-text matching model, thereby continuously improving the accuracy of the annotation results.
[0100] Based on this, since the annotation result can be displayed in the form of a segmentation heat map, and the segmentation heat map can include multiple pixels, in order to ensure the accuracy of the cropped third image, in some embodiments, the above-mentioned cropping of the first image according to pixels can specifically include:
[0101] Segment the segmentation heat map according to multiple pixels to obtain multiple regions;
[0102] mapping a plurality of regions into the first image;
[0103] The first image is cropped according to a plurality of regions.
[0104] Here, since the image segmentation algorithm can be used to locate objects and boundaries (lines, curves, etc.) in the image, each pixel can correspond to a portion of the area in the segmentation heat map. That is, the segmentation heat map can include multiple areas. Since the segmentation heat map is obtained by performing image segmentation on the first image, the first image can include multiple areas that are the same as the multiple areas in the segmentation heat map. Since pixels can correspond one-to-one to objects, and pixels can correspond one-to-one to areas, objects can correspond one-to-one to areas. Therefore, cropping the first image according to pixels can be cropping the first image according to areas.
[0105] Since the first image and the segmentation heat map have the same size, by mapping multiple regions into the first image, it can be ensured that the multiple regions in the first image are the same as the multiple regions in the segmentation heat map, thereby ensuring the accuracy of the cropped third image.
[0106] In actual situations, if the pixel boundaries are complex, it will increase the difficulty of image clipping and reduce the efficiency of image clipping. Therefore, in order to improve the efficiency of image clipping, in some embodiments, the segmentation of the segmentation heat map according to multiple pixels may specifically include:
[0107] Determine the minimum bounding rectangle corresponding to each of the multiple pixels;
[0108] The segmentation heat map is segmented according to multiple minimum bounding rectangles.
[0109] Here, since multiple pixel points can form pixels corresponding to an object, there can be a corresponding relationship between pixel points and pixels. After obtaining multiple pixel points of an object, the minimum bounding rectangle of the multiple pixel points can be determined by existing methods. The minimum bounding rectangle of the multiple pixel points can be the minimum bounding rectangle corresponding to the pixels representing the object.
[0110] Since the boundary of the rectangle is simple, segmenting the segmented heat map by using a plurality of minimum circumscribed rectangles and then cropping the first image according to the rectangles can improve the efficiency of image cropping.
[0111] In order to better describe the entire solution, some specific examples are given based on the above embodiments.
[0112] For example, Figure 2 As shown, after the first image is input into the self-supervised large model, the self-supervised large model can extract features from the first image to obtain image features of the first image. After the image features are input into the target downstream task head, the target downstream task head can annotate the image features corresponding to the target annotation task to obtain an annotation result. After obtaining the annotation result, the first target annotation result and / or the second target annotation result can be obtained from the annotation result. Then, the target downstream task head is iteratively optimized using the first target annotation result and / or the second target annotation result. Among them, the iterative optimization can be performed once every preset time length.
[0113] Specifically, when the target downstream task head is used to process an image classification task, the labeling result may be a category label corresponding to the first image. The first target labeling result may be a labeling result in which the first confidence of the category label is greater than a first preset threshold. After obtaining the first target labeling result, the first target labeling result may be used to iteratively optimize the target downstream task head.
[0114] In the case where the target downstream task head is used to process the target detection task, the annotation result may include the category and detection box of the object in the first image. The first target annotation result may be an annotation result in which the second confidence of the detection box is greater than the second preset threshold. After obtaining the first target annotation result, the image and category of an object in the first target annotation result may be input into the large image-text matching model together, and the image and category text of the object may be matched using the large image-text matching model to obtain a first matching degree. In the case where the first matching degree is greater than the third preset threshold, the first target annotation result may be determined as the second target annotation result. The target downstream task head and the large image-text matching model may be iteratively optimized using the second target annotation result.
[0115] In the case where the target downstream task head is used to process an image segmentation task, the annotation result may include the category and pixels of the object in the first image. The first target annotation result may be an annotation result in which the third confidence level of the pixel is greater than a fourth preset threshold. After obtaining the first target annotation result, the image and category of an object in the first target annotation result may be input into the large image-text matching model together, and the image and category text of the object may be matched using the large image-text matching model to obtain a second matching degree. When the second matching degree is greater than the fifth preset threshold, the first target annotation result may be determined as the second target annotation result. The target downstream task head and the large image-text matching model may be iteratively optimized using the second target annotation result.
[0116] Therefore, by using the target downstream task head to annotate the image features output by the self-supervised large model instead of the first image corresponding to the target annotation task, the annotation process of the target downstream task head can be simplified, the annotation efficiency of the target downstream task head can be improved, and the overall annotation efficiency of automatic annotation can be improved. By continuously iteratively optimizing the target downstream task head, the processing performance of the target downstream task head can be continuously improved, and the accuracy of the annotation results can be continuously improved.
[0117] Based on the image annotation method provided in the above embodiment, the present application also provides a specific implementation of the image annotation device. Please refer to the following embodiment.
[0118] like Figure 3 As shown, the image annotation device 300 provided in the embodiment of the present application includes the following modules:
[0119] A first acquisition module 310, used to acquire a first image to be labeled;
[0120] An extraction module 320, configured to extract features from the first image using a self-supervised large model to obtain image features of the first image;
[0121] The labeling module 330 is used to use the target downstream task head to perform image labeling corresponding to the target labeling task on the image features to obtain the labeling result of the first image. The target labeling task is any one of multiple computer vision labeling tasks, and the target downstream task head is a neural network structure for processing the target labeling task.
[0122] The image annotation device 300 is described in detail below, as follows:
[0123] In some embodiments, the target annotation task includes an image classification task, and the annotation result includes a category label corresponding to the first image. Based on this, the image annotation device 300 may further include:
[0124] A second acquisition module, configured to acquire a first confidence level corresponding to the category label after obtaining the labeling result of the first image;
[0125] The first optimization module is used to iteratively optimize the target downstream task head using the first image and its corresponding category label when the first confidence level is greater than a first preset threshold.
[0126] In some embodiments, the target annotation task includes a target detection task, and the annotation result includes the category and detection box of each object in the first image. Based on this, the image annotation device 300 may also include:
[0127] A third acquisition module is used to obtain a second confidence level corresponding to each detection frame after obtaining the annotation result of the first image;
[0128] A first cropping module, configured to crop the first image according to the detection frame to obtain at least one second image when the second confidence level is greater than a second preset threshold;
[0129] A first matching module is used to perform image-text matching on the second image and the category text of the second image using the image-text matching large model for each second image, so as to obtain a first matching degree;
[0130] The second optimization module is used to iteratively optimize the target downstream task head by using the second image and its corresponding category when the first matching degree is greater than a third preset threshold.
[0131] In some embodiments, the target annotation task includes an image segmentation task, and the annotation result includes the category and pixels of each object in the first image. Based on this, the image annotation device 300 may also include:
[0132] A fourth acquisition module, configured to acquire a third confidence level corresponding to each pixel after obtaining the annotation result of the first image;
[0133] A second cropping module, configured to crop the first image by pixel to obtain a plurality of third images when the third confidence level is greater than a fourth preset threshold;
[0134] A second matching module is used to perform image-text matching on each third image and the category text of the third image using the image-text matching large model to obtain a second matching degree;
[0135] The third optimization module is used to iteratively optimize the target downstream task head by using the third image and its corresponding category when the second matching degree is greater than a fifth preset threshold.
[0136] In some embodiments, the annotation result is a segmentation heat map, and the segmentation heat map includes a plurality of pixels. Based on this, the second clipping module may specifically include:
[0137] The segmentation submodule is used to segment the segmentation heat map according to multiple pixels to obtain multiple regions;
[0138] A mapping submodule, used for mapping a plurality of regions into a first image;
[0139] The cropping submodule is used to crop the first image according to multiple regions.
[0140] In some embodiments, the segmentation submodule may specifically include:
[0141] A determination unit, used to determine the minimum bounding rectangle corresponding to each of the plurality of pixels;
[0142] The segmentation unit is used to segment the segmentation heat map according to multiple minimum bounding rectangles.
[0143] In some embodiments, the image annotation apparatus 300 may further include:
[0144] a fifth acquisition module, configured to acquire a training data set according to a preset period after obtaining the annotation result of the first image, wherein the training data set includes images of multiple vehicle cockpits;
[0145] The fourth optimization module is used to iteratively optimize the self-supervised large model using the training data set.
[0146] In the image annotation device of the embodiment of the present application, since the self-supervised large model is a large model with strong generalization ability, there is no need to perform additional training on the self-supervised large model, and the self-supervised large model can be used to extract features from the first image to obtain the image features of the first image. Since there is no need to perform additional training on the large model, it is possible to save time for model training and improve annotation efficiency. By using the target downstream task head to annotate the image features instead of the first image with the target annotation task, the annotation process of the target downstream task head can be simplified and the annotation efficiency of the target downstream task head can be improved. In this way, through the embodiment of the present application, the overall annotation efficiency of automatic annotation can be improved.
[0147] Based on the image annotation method provided in the above embodiment, the embodiment of the present application also provides a specific implementation of the electronic device. Figure 4 A schematic diagram of an electronic device 400 provided in an embodiment of the present application is shown.
[0148] The electronic device 400 may include a processor 410 and a memory 420 storing computer program instructions.
[0149] Specifically, the processor 410 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0150] The memory 420 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 420 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 420 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 420 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 420 is a non-volatile solid-state memory.
[0151] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present application.
[0152] The processor 410 implements any one of the image annotation methods in the above embodiments by reading and executing computer program instructions stored in the memory 420 .
[0153] In one example, the electronic device 400 may further include a communication interface 430 and a bus 440. Figure 4 As shown, the processor 410, the memory 420, and the communication interface 430 are connected via a bus 440 and communicate with each other.
[0154] The communication interface 430 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0155] Bus 440 includes hardware, software or both, and the parts of electronic equipment are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 440 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.
[0156] Exemplarily, the electronic device 400 may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA).
[0157] The electronic device can execute the image annotation method in the embodiment of the present application, thereby realizing the combination Figures 1 to 3 The image annotation method and device are described.
[0158] In addition, in combination with the image annotation method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the image annotation methods in the above embodiment is implemented.
[0159] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0160] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0161] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0162] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.
[0163] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. An image annotation method, characterized in that: include: Acquire a first image to be annotated; Extracting features from the first image using a self-supervised large model to obtain image features of the first image; The target downstream task head is used to perform image annotation corresponding to a target annotation task on the image features to obtain an annotation result of the first image, wherein the target annotation task is any one of a plurality of computer vision annotation tasks, and the target downstream task head is a neural network structure for processing the target annotation task.
2. The method according to claim 1, characterized in that The target labeling task includes an image classification task, the labeling result includes a category label corresponding to the first image, and after obtaining the labeling result of the first image, the method further includes: Obtaining a first confidence level corresponding to the category label; When the first confidence level is greater than a first preset threshold, the target downstream task head is iteratively optimized using the first image and its corresponding category label.
3. The method according to claim 1, characterized in that The target labeling task includes a target detection task, and the labeling result includes a category and a detection box of each object in the first image. After obtaining the labeling result of the first image, the method further includes: Obtaining a second confidence level corresponding to each of the detection frames; When the second confidence level is greater than a second preset threshold, cropping the first image according to the detection frame to obtain at least one second image; For each of the second images, use a large image-text matching model to perform image-text matching on the second image and the category text of the second image to obtain a first matching degree; When the first matching degree is greater than a third preset threshold, the target downstream task head is iteratively optimized using the second image and its corresponding category.
4. The method according to claim 1, characterized in that: The target labeling task includes an image segmentation task, and the labeling result includes the category and pixel of each object in the first image. After obtaining the labeling result of the first image, the method further includes: Obtaining a third confidence level corresponding to each of the pixels; When the third confidence level is greater than a fourth preset threshold, cropping the first image according to the pixels to obtain a plurality of third images; For each of the third images, use a large image-text matching model to perform image-text matching on the third image and the category text of the third image to obtain a second matching degree; When the second matching degree is greater than a fifth preset threshold, the target downstream task head is iteratively optimized using the third image and its corresponding category.
5. The method according to claim 4, characterized in that The annotation result is a segmentation heat map, the segmentation heat map includes a plurality of pixels, and the cropping of the first image according to the pixels includes: Segment the segmentation heat map according to the multiple pixels to obtain multiple regions; mapping the plurality of regions into the first image; The first image is cropped according to the multiple regions.
6. The method according to claim 5, characterized in that The segmenting the segmentation heat map according to the plurality of pixels includes: Determine the minimum bounding rectangle corresponding to each of the plurality of pixels; The segmentation heat map is segmented according to a plurality of the minimum circumscribed rectangles.
7. The method according to claim 1, characterized in that After obtaining the annotation result of the first image, the method further includes: Acquire a training data set according to a preset period, wherein the training data set includes images of multiple vehicle cockpits; The self-supervised large model is iteratively optimized using the training data set.
8. An image annotation device, characterized in that: The device comprises: A first acquisition module, used to acquire a first image to be labeled; An extraction module, used for performing feature extraction on the first image using a self-supervised large model to obtain image features of the first image; A labeling module is used to use a target downstream task head to perform image labeling corresponding to a target labeling task on the image features to obtain a labeling result of the first image, wherein the target labeling task is any one of a plurality of computer vision labeling tasks, and the target downstream task head is a neural network structure for processing the target labeling task.
9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the image annotation method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the image annotation method according to any one of claims 1 to 7 is implemented.