Image labeling method and device, storage medium and computer program product

Abstract

The invention discloses an image annotation method and device, a storage medium and a computer program product, and the method comprises the steps: obtaining the initial bounding box information of an interested target in a to-be-annotated first image, and enabling the initial bounding box information to be obtained based on the annotation of an annotation person; generating, by an image encoder of an image segmentation model, an image embedding vector based on the first image; generating a sparse embedding vector based on the initial bounding box information through a prompt encoder of an image segmentation model; and generating, by a mask decoder of an image segmentation model, a mask of the target of interest in the first image based on the image embedding vector and the sparse embedding vector, and labeling the target of interest in the first image based on the mask of the target of interest in the first image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image annotation method, device, computing device, storage medium and computer program product. Background Art

[0002] Artificial Intelligence (AI) detection or segmentation technology is widely used in machine vision, which can detect objects of interest. For example, object detection or semantic segmentation can detect defects in product appearance, so as to monitor product production quality in a timely manner and ensure that products meet quality standards and customer requirements.

[0003] In related technologies, methods for detecting objects of interest usually rely on labelers to manually label the areas of interest in product appearance images, and achieve detection of objects of interest through model training, such as defect detection, package segmentation, etc.

[0004] However, labelers face many challenges in the actual labeling process. These challenges are mainly reflected in the fact that labeling work is not only time-consuming and laborious, but also easily affected by the subjective factors of the labelers, resulting in inconsistency in the labeling results. Especially in the process of multi-person labeling, due to the different experience and judgment standards of different labelers, the accuracy and consistency of the labeling results are difficult to guarantee. In particular, it is difficult to achieve high-precision labeling when facing small defects or complex backgrounds. Summary of the invention

[0005] The embodiments of the present application provide an image annotation method, device, computing device, storage medium and computer program product to solve the problem that the annotation methods in the related art are difficult to achieve high-precision annotation.

[0006] The present application embodiment adopts the following technical solutions: In a first aspect, the present application provides an image annotation method, comprising: Acquire initial bounding box information of an object of interest in a first image to be annotated, wherein the initial bounding box information is obtained based on annotations by an annotator; Generate an image embedding vector based on the first image through an image encoder of an image segmentation model; Generate a sparse embedding vector based on the initial bounding box information through a hint encoder of an image segmentation model; A mask of the target of interest in the first image is generated based on the image embedding vector and the sparse embedding vector by a mask decoder of the image segmentation model, and the target of interest in the first image is labeled based on the mask of the target of interest in the first image.

[0007] In a second aspect, the present application provides an image annotation device, comprising: An information acquisition module, used to acquire initial bounding box information of an object of interest in a first image to be annotated, wherein the initial bounding box information is obtained based on annotations by annotators; A first generating module, configured to generate an image embedding vector based on the first image by using an image encoder of an image segmentation model; A second generation module is used to generate a sparse embedding vector based on the initial bounding box information through a hint encoder of the image segmentation model; An image annotation module is used to generate a mask of the object of interest in the first image based on the image embedding vector and the sparse embedding vector through a mask decoder of the image segmentation model, and to annotate the object of interest in the first image based on the mask of the object of interest in the first image.

[0008] In a third aspect, the present application provides a computing device, including: a memory and a processor; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: Acquire initial bounding box information of an object of interest in a first image to be annotated, wherein the initial bounding box information is obtained based on annotations by an annotator; Generate an image embedding vector based on the first image through an image encoder of an image segmentation model; Generate a sparse embedding vector based on the initial bounding box information through a hint encoder of an image segmentation model; A mask of the target of interest in the first image is generated based on the image embedding vector and the sparse embedding vector by a mask decoder of the image segmentation model, and the target of interest in the first image is labeled based on the mask of the target of interest in the first image.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the image annotation method described in the first aspect above are implemented.

[0010] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the image annotation method described in the first aspect above.

[0011] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: By adopting the image annotation method provided in the embodiment of the present application, it is possible to obtain the initial bounding box information of the target of interest obtained by the annotation personnel in the first image to be annotated, and generate an image embedding vector based on the first image through the image encoder of the image segmentation model, and generate a sparse embedding vector based on the initial bounding box information through the prompt encoder of the image segmentation model, and finally, the mask of the target of interest in the first image can be generated by using the mask decoder of the image segmentation model with the sparse embedding vector as the prompt information of the target of interest, and searching in the image embedding vector, and finally based on the mask of the target of interest in the first image, it is possible to achieve accurate annotation of the target of interest in the first image. Since the image segmentation model can be used to predict a more accurate mask of the target of interest in the first image with the initial bounding box information as the prompt information, the annotation information finally determined based on the mask of the target of interest is more in line with the shape of the target of interest, which ensures the accuracy of the annotation results, and at the same time, it can also reduce the reliability of the annotation results affected by the different experience and judgment criteria of the annotation personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of an implementation flow of an image annotation method is provided for an embodiment of the present application; Figure 2 A schematic diagram of a process for determining an annotation type according to a task category in an image annotation method provided in an embodiment of the present application; Figure 3 A schematic diagram of an annotation interface in an image annotation method provided in an embodiment of the present application; Figure 4 A schematic diagram of the mask effect of segmentation task annotation in the image annotation method provided in an embodiment of the present application; Figure 5 The image annotation method provided in the embodiment of the present application is based on Figure 4 Schematic diagram of the minimum bounding rectangle generated by the mask in; Figure 6 A schematic diagram of the annotation results in the target detection task in the image annotation method provided in an embodiment of the present application; Figure 7 A schematic diagram of an implementation flow of predicting annotation results by using an image segmentation model in the image annotation method provided in an embodiment of the present application; Figure 8 A schematic diagram of a process for fine-tuning manual annotation results by using an image segmentation model in the image annotation method provided in an embodiment of the present application; Fig. 9A schematic diagram of the correction result of the annotation result of the segmentation task in the image annotation method provided in an embodiment of the present application; Fig.10 A schematic diagram of the correction result of the annotation result of the target detection task in the image annotation method provided in the embodiment of the present application; Fig.11 A schematic diagram showing the visualization of the correction effect of annotation for target detection tasks in the image annotation method provided in an embodiment of the present application; Fig.12 A schematic diagram showing a visualization of the annotation correction effect of another target detection task in the image annotation method provided in an embodiment of the present application; Fig.13 A schematic diagram showing the visualization of the annotation correction effect of the image segmentation task in the image annotation method provided in the embodiment of the present application; Fig.14 A schematic diagram of the structure of an image annotation device is provided for an embodiment of the present application; Fig.15 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0014] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0016] First, the terms involved in one or more embodiments of the present application are explained.

[0017] ROI, the full name of which is Region of Interest, is also called Region of Interest in Chinese. In deep learning, a common application is to use convolutional neural networks for object detection. When processing images or videos, the network needs to identify objects in the image, and usually, it is only interested in some specific areas of the image, rather than the entire image. This specific area can be called ROI.

[0018] Mask, also known as mask in English, uses selected images, graphics or objects to block the processed image (all or part) to control the area or process of image processing. The specific image or object used for covering is called a mask or template. Mask is a binary image composed of 0 and 1. When a mask is applied in a certain function, the 1-value area is processed, and the masked 0-value area is not included in the calculation. Image masks are defined by specifying data values, data ranges, finite or infinite values, regions of interest, and annotation files. Masks can also be established by applying any combination of the above options as input. Masks are templates for image filters. For example, in product quality inspection under machine vision scenarios, when extracting surface defects of products, an N*N matrix is ​​used to filter pixels of the image, and then the marks we need are highlighted. This matrix is ​​a kind of mask.

[0019] Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing targets of interest. It is a key step from image processing to image analysis. Existing image segmentation methods are mainly divided into the following categories: threshold-based segmentation methods, region-based segmentation methods, edge-based segmentation methods, and segmentation methods based on specific theories, such as deep learning. From a mathematical point of view, image segmentation is the process of dividing a digital image into non-overlapping regions. The image segmentation process is also a labeling process, that is, pixels belonging to the same region are assigned the same number.

[0020] Semantic segmentation: Semantic segmentation is a classification at the pixel level. Pixels belonging to the same category are classified into one category. Therefore, semantic segmentation understands images at the pixel level. For example, in an image, pixels belonging to people are classified into one category, pixels belonging to motorcycles are also classified into one category, and background pixels are also classified into one category.

[0021] In order to solve the above problems existing in the prior art, the embodiments of the present application provide an image annotation method. Specifically, in the present application, an image annotation method is provided, and the present application also relates to an image annotation device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0022] The execution subject of the method may be various types of computing devices, or may be an application or application (Application, APP) installed on the computing device, such as a package grabbing application. The computing device may be, for example, a user terminal such as a mobile phone, a tablet computer, a smart wearable device, or a server.

[0023] For ease of description, the present application embodiment takes the execution subject of the method as an example of a server to introduce the method. Those skilled in the art can understand that the present application embodiment takes the server as an example to introduce the method, which is only an exemplary description and does not limit the scope of protection of the claims corresponding to the present solution.

[0024] Specifically, a schematic diagram of an implementation flow of an image annotation method provided by one or more embodiments of the present application is as follows: Figure 1 As shown, including: Step 110 , obtaining initial bounding box information of the object of interest in the first image to be annotated, where the initial bounding box information is obtained based on annotations by annotators.

[0025] The initial bounding box information can be divided into a coarse-grained mask and a coarse-grained box according to the task category of the first image. When the task category of the first image is a segmentation task, the initial bounding box information is a coarse-grained mask, and when the task category of the first image is an object detection task, the initial bounding box information is a coarse-grained box.

[0026] Optionally, to avoid low image quality, such as the presence of reflection, noise, occlusion, blur and other adverse factors in the image, which leads to low image quality and thus affects the image annotation effect, the embodiment of the present application can judge the clarity of the image before the annotator annotates the target of interest in the first image, and when it is determined that the clarity of the image does not meet the set clarity condition, perform image enhancement processing on the first image. Specifically, obtaining the initial bounding box information of the target of interest in the first image to be annotated includes: When the clarity of the first image to be annotated is less than or equal to a preset clarity threshold, performing image enhancement on the first image so that the clarity of the first image after image enhancement is greater than the preset clarity threshold; Obtain initial bounding box information of the object of interest in the first image after image enhancement.

[0027] Among them, the image enhancement of the first image can be achieved by using the image enhancement technology in the related art, and the embodiment of the present application does not make any specific limitation on this.

[0028] Optionally, the first image can be used for a segmentation task or a target detection task. Specifically, obtaining initial bounding box information of a target of interest in the first image to be labeled includes: When the first image to be annotated is used for a segmentation task, mask annotation information of an object of interest in the first image by an annotator is obtained; and a minimum circumscribed rectangular frame is generated based on the mask annotation information as initial bounding box information of the object of interest in the first image; When the first image to be annotated is used for a target detection task, bounding box annotation information of the target of interest in the first image by the annotator is obtained as initial bounding box information of the target of interest in the first image.

[0029] Figure 2 A schematic diagram of a process for determining the annotation type according to the task category in the image annotation method provided in the embodiment of the present application. Figure 2 In the example, assuming that the clarity of the first image is less than or equal to the preset clarity threshold, the first image can be enhanced first, and then the annotation type can be determined according to the task category of the first image. When the task category of the first image is determined to be a segmentation task, the annotator annotates the first image to obtain a coarse-grained mask; when the task category of the first image is determined to be an object detection task, the annotator annotates the first image to obtain a coarse-grained frame.

[0030] Among them, the task categories include segmentation tasks and target detection tasks. The segmentation task (English name is Segmentation) is to refine the annotation box based on the detection task, and to classify each pixel. The segmentation task is a supervised learning method that divides data into multiple different categories based on label information. In deep learning, the core concepts involved in the segmentation task include clustering and segmentation. Clustering is an unsupervised learning method that automatically divides data into multiple clusters based on the similarity between data. For example, a picture includes people, sheep, and dogs. The segmentation task is to classify which pixels in the picture belong to which object, that is, to classify the pixels in the picture. The task of the target detection task (English name is Object Detection) is to find all the targets (objects) of interest in the image and determine their categories and locations. It is one of the core problems in the field of computer vision. Unlike the segmentation task, the annotation result of the target detection task is a detection box, while the annotation result of the segmentation task is in the form of an image mask. Figure 3 This is a schematic diagram of the annotation interface in the image annotation method provided in the embodiment of the present application. Figure 3 Manually select the annotation area and annotation category in .

[0031] Specifically, ① if it is a segmentation task, first Figure 4 FIG. 1 is a schematic diagram of a mask effect of segmentation task annotation in the image annotation method provided in an embodiment of the present application. Figure 4 The mask information of the defect (the target of interest in the image) can be obtained by manual annotation by annotators. Figure 5The image annotation method provided in the embodiment of the present application is based on Figure 4 Schematic diagram of the minimum bounding rectangle generated by the mask in Figure 4 The minimum bounding rectangle generated by the mask information shown is the initial bounding box information of the defect (i.e., the target of interest in the figure).

[0032] ② If it is a target detection task, such as Figure 6 FIG. 1 is a schematic diagram of the labeling result of the target detection task in the image labeling method provided in the embodiment of the present application. Figure 6 The annotator can directly obtain the initial bounding box information of the defect (i.e. the target of interest in the image). Figure 4 and Figure 6 It can be clearly observed that the inconsistency between the annotator’s experience and the annotation standards will lead to poor annotation accuracy (i.e., the initial bounding boxes do not fit the actual defect area).

[0033] Step 120: Generate an image embedding vector based on the first image through an image encoder of the image segmentation model.

[0034] The image segmentation network may include three modules, namely, an image encoder, a hint encoder and a mask decoder. The image encoder is intended to map the first image to be annotated to the image feature space. The hint encoder is responsible for mapping the input hint information to the feature space of the hint information, wherein the hint information may include three forms, the first is the hint information in the form of points, which is used to hint the position and form of the target of interest that needs to be annotated (i.e., segmentation or target detection) in the first image to be annotated in the form of points, the second is the hint information in the form of a target frame, which is used to hint the position and form of the target of interest that needs to be annotated in the first image to be annotated in the form of a target frame, and the third is the hint information in the form of text, which is used to hint the description of the target of interest that needs to be annotated in the first image to be annotated in the form of text. The mask decoder has two main functions, one is to integrate the two feature vector embeddings output by the image encoder and the hint encoder respectively (wherein, the feature vector output by the image encoder is the image embedding vector, and the feature vector output by the hint encoder is the sparse embedding vector), and then decode the feature map feature map of the two feature vector embeddings to obtain the final image annotation result of the first image.

[0035] Optionally, generating an image embedding vector based on the first image by an image encoder of the image segmentation model includes: Resizing the first image to a specified size to obtain a first image of the specified size; The image encoder of the image segmentation model obtains a feature vector of the first image, and an absolute position code and a relative position code of the first image based on the first image of a specified size; The image encoder of the image segmentation model generates an image embedding vector based on the feature vector of the first image, and the absolute position code and the relative position code of the first image.

[0036] Among them, relative position encoding and absolute position encoding are two different position encoding methods. Absolute position encoding is a method based on position embedding, in which each position is assigned a unique position vector. These vectors are fixed and have nothing to do with the content of the input sequence. This encoding method works well for processing shorter sequences, but may have problems when processing long sequences because the length of the sequence exceeds the range of position encoding that the model can handle. Relative position encoding is a method based on relative position, in which each position is encoded as an offset, indicating the relative distance between the position and other positions. Relative position encoding can be achieved by adding additional information to the input embedding. This encoding method can handle long sequences and can maintain a certain consistency in different contexts.

[0037] The embodiment of the present application combines the characteristics of relative position coding and absolute position coding, extracts the absolute position coding and relative position coding of the first image through the image encoder of the image segmentation model, and combines the feature vector of the first image, as well as the absolute position coding and relative position coding of the first image through the image encoder of the image segmentation model to generate an image embedding vector.

[0038] As an example, the process of generating an image embedding vector may include: S3.1, obtaining the image patch. Specifically, the first image may be resized to a square and its size may be adjusted. After the resized first image is input to the image encoder of the image segmentation model, a convolution operation is first performed using the convolution kernel of the image encoder to generate a feature map, and then an absolute positional embedding is added to the feature map.

[0039] S3.2, obtain the relative positional embedding. In the attention module of the image encoder, operate the attention matrix of the feature map Feature Map, generate a relative position encoding vector and add it to the attention matrix of the feature map Feature Map.

[0040] S3.3, generating an image embedding vector through the Neck module of the image encoder. The Neck module consists of two convolutional layers and is used to generate an image embedding vector ImageEmbedding based on the feature map with absolute position encoding and relative position encoding.

[0041] Step 130 , generating a sparse embedding vector based on the initial bounding box information through a hint encoder of the image segmentation model.

[0042] Among them, the hint encoder generates a sparse embedding vector based on the initial bounding box information. Specifically, the input initial bounding box information is mapped to the feature space of the hint information to obtain a sparse embedding vector.

[0043] Figure 7 This is a schematic diagram of the implementation process of predicting the annotation results through the image segmentation model in the image annotation method provided in the embodiment of the present application. Figure 7 In the method, the first image is input into the image encoder of the image segmentation model, and the image encoder processes the first image to obtain an image embedding vector Image embedding; and the initial boundary information of the first image is input into the hint encoder of the image segmentation model, and the hint encoder generates a sparse embedding vector based on the initial bounding box information; finally, the image embedding vector and the sparse embedding vector are both input into the mask decoder of the image segmentation model, and the mask decoder decodes them to predict the mask of the target of interest in the first image.

[0044] Step 140, generating a mask of the object of interest in the first image based on the image embedding vector and the sparse embedding vector through a mask decoder of the image segmentation model, and labeling the object of interest in the first image based on the mask of the object of interest in the first image.

[0045] The mask decoder of the image segmentation model is used to perform decoding operations on the image embedding vector and the sparse embedding vector to obtain a mask of the object of interest in the first image.

[0046] Figure 8A flow chart of fine-tuning the manual annotation results through the image segmentation model in the image annotation method provided in the embodiment of the present application. Since the annotation result of the segmentation task is in the form of an image mask, the annotation result of the target detection task is in the form of a detection box. Therefore, after obtaining the mask of the target of interest predicted by the image segmentation model, the initial bounding box information can be fine-tuned based on the mask of the target of interest according to the task type of the first image. Specifically, when it is determined that the task type of the first image is a segmentation task, the mask of the target of interest in the first image can be mapped to the first image before the specified size adjustment, and the mapping result of the mask of the target of interest in the first image before the specified size adjustment is used as the annotation result of the target of interest in the first image (i.e., the annotation result after fine-tuning in the figure). When it is determined that the task type of the first image is a target detection task, the mask of the target of interest in the first image can be mapped to the first image before the specified size adjustment, and based on the mapping result of the mask of the target of interest in the first image before the specified size adjustment, the position information of the minimum circumscribed rectangle of the contour of the mask of the target of interest is determined. Finally, based on the position information of the minimum circumscribed rectangle of the contour of the mask of the target of interest, the target of interest in the first image is marked.

[0047] Optionally, since the image annotation result in the segmentation task is also in the form of an image mask, the mask of the object of interest in the first image can be mapped to the original first image to obtain the first image as the image annotation result of the segmentation task. Specifically, based on the mask of the object of interest in the first image, annotating the object of interest in the first image includes: When the first image to be annotated is used for a segmentation task, mapping a mask of an object of interest in the first image to the first image before a specified size adjustment; The mapping result of the mask of the object of interest in the first image before the specified size adjustment is used as the labeling result of the object of interest in the first image.

[0048] Optionally, since the image annotation result in the target detection task is in the form of a detection frame, after obtaining the mask of the target of interest in the first image, it is necessary to further process it to obtain the minimum bounding rectangle of the mask after the mask is mapped to the original first image, as the image annotation result of the first image segmentation task. Based on the mask of the target of interest in the first image, annotating the target of interest in the first image includes: When the first image to be annotated is used for an object detection task, mapping a mask of an object of interest in the first image to the first image before a specified size adjustment; Determine position information of a minimum circumscribed rectangle of an outline of the mask of the object of interest based on a mapping result of the mask of the object of interest in the first image before the specified size adjustment; The object of interest in the first image is marked based on the position information of the minimum circumscribed rectangle of the outline of the mask of the object of interest.

[0049] As an example, according to different tasks: ① split the task: Fig. 9 The figure shows a schematic diagram of the correction result of the annotation result of the segmentation task in the image annotation method provided in the embodiment of the present application. The defect mask (i.e., the mask of the target of interest) generated in the refined segmentation stage can be directly mapped back to the original image to obtain the final annotation correction result diagram. ② Target detection task: First, the mask generated by the refined segmentation stage (i.e., the mask of the target of interest) quickly finds the defect contour, traverses all contours, and then calculates the minimum circumscribed rectangular frame of each contour, and returns the coordinates of the upper left corner of the contour and the width and height of the rectangular frame. Finally, as Fig.10 The figure shows a schematic diagram of the correction result of the annotation result of the target detection task in the image annotation method provided in an embodiment of the present application. A minimum circumscribed rectangular box can be drawn on the image and mapped back to the original image to obtain the final annotation correction result diagram. Fig.10 The schematic diagram of the correction result of the annotation result of the target detection task in the image annotation method provided in the embodiment of the present application shows that the rectangular box generated by the mask of the target of interest predicted by the image segmentation model is more accurate and fits the defect better than the manually annotated bounding box. Fig.11 A visualization diagram of the correction effect of the target detection task annotation in the image annotation method provided in the embodiment of the present application, in Fig.11 In the example, the image annotation result is finally in the form of an annotation box, and the final corrected annotation box is closer to the actual size of the original object of interest than the initial annotation box. Fig.12 A visualization diagram of the annotation correction effect of another target detection task in the image annotation method provided in an embodiment of the present application, Fig.12 In the example, the image annotation result is also in the form of an annotation box, and the final corrected annotation box is closer to the actual size of the original object of interest than the initial annotation box. Fig.13 A visualization diagram of the correction effect of image segmentation task annotation in the image annotation method provided in the embodiment of the present application, Fig.13 In the above, the image annotation result is finally in the form of an image mask, and the final corrected image mask is closer to the actual size of the original object of interest than the initial image mask.

[0050] By adopting the image annotation method provided in the embodiment of the present application, it is possible to obtain the initial bounding box information of the target of interest obtained by the annotation personnel in the first image to be annotated, and generate an image embedding vector based on the first image through the image encoder of the image segmentation model, and generate a sparse embedding vector based on the initial bounding box information through the prompt encoder of the image segmentation model, and finally, the mask of the target of interest in the first image can be generated by using the mask decoder of the image segmentation model with the sparse embedding vector as the prompt information of the target of interest, and searching in the image embedding vector, and finally based on the mask of the target of interest in the first image, it is possible to achieve accurate annotation of the target of interest in the first image. Since the image segmentation model can be used to predict a more accurate mask of the target of interest in the first image with the initial bounding box information as the prompt information, the annotation information finally determined based on the mask of the target of interest is more in line with the shape of the target of interest, which ensures the accuracy of the annotation results, and at the same time, it can also reduce the reliability of the annotation results affected by the different experience and judgment criteria of the annotation personnel.

[0051] Fig.14 FIG. 1 is a schematic diagram of a structure of an image annotation device 1400 provided in an exemplary embodiment of the present application. Fig.14 As shown, the device 1400 includes: an information acquisition module 1410, a first generation module 1420, a second generation module 1430 and an image annotation module 1440, wherein: An information acquisition module 1410 is used to acquire initial bounding box information of an object of interest in a first image to be annotated, wherein the initial bounding box information is obtained based on annotations by annotators; A first generating module 1420, configured to generate an image embedding vector based on the first image by using an image encoder of an image segmentation model; A second generation module 1430 is used to generate a sparse embedding vector based on the initial bounding box information through a hint encoder of the image segmentation model; The image annotation module 1440 is used to generate a mask of the target of interest in the first image based on the image embedding vector and the sparse embedding vector through a mask decoder of the image segmentation model, and to annotate the target of interest in the first image based on the mask of the target of interest in the first image.

[0052] By using the device provided in the embodiment of the present application, it is possible to obtain the initial bounding box information of the target of interest obtained by the annotator in the first image to be annotated, and to generate an image embedding vector based on the first image through the image encoder of the image segmentation model, and to generate a sparse embedding vector based on the initial bounding box information through the prompt encoder of the image segmentation model, and finally to generate a mask of the target of interest in the first image by using the sparse embedding vector as the prompt information of the target of interest through the mask decoder of the image segmentation model, and finally to achieve accurate annotation of the target of interest in the first image based on the mask of the target of interest in the first image. Since the image segmentation model can be used to predict a more accurate mask of the target of interest in the first image using the initial bounding box information as the prompt information, the annotation information finally determined based on the mask of the target of interest is more in line with the shape of the target of interest, thereby ensuring the accuracy of the annotation results, and at the same time, it can also reduce the reliability of the annotation results affected by the different experience and judgment criteria of the annotators.

[0053] In some exemplary embodiments, when the information acquisition module 1410 acquires the initial bounding box information of the object of interest in the first image to be annotated, it is specifically used to: When the first image to be annotated is used for a segmentation task, obtaining mask annotation information of an object of interest in the first image by an annotator; and generating a minimum circumscribed rectangular frame based on the mask annotation information as initial bounding box information of the object of interest in the first image; When the first image to be annotated is used for a target detection task, bounding box annotation information of the target of interest in the first image by the annotator is obtained as initial bounding box information of the target of interest in the first image.

[0054] In some exemplary embodiments, when the first generating module 1420 generates an image embedding vector based on the first image through an image encoder of an image segmentation model, it is specifically used to: Resizing the first image to a specified size to obtain a first image of the specified size; Obtaining, by an image encoder of an image segmentation model, a feature vector of the first image, and an absolute position code and a relative position code of the first image based on a first image of a specified size; The image encoder of the image segmentation model generates the image embedding vector based on the feature vector of the first image, and the absolute position encoding and relative position encoding of the first image.

[0055] In some exemplary embodiments, when the information acquisition module 1410 acquires the initial bounding box information of the object of interest in the first image to be annotated, it is specifically used to: When the clarity of the first image to be marked is less than or equal to a preset clarity threshold, performing image enhancement on the first image so that the clarity of the first image after image enhancement is greater than the preset clarity threshold; Acquire initial bounding box information of the object of interest in the first image after the image enhancement.

[0056] In some exemplary embodiments, when the image annotation module 1440 annotates the object of interest in the first image based on the mask of the object of interest in the first image, it is specifically used to: When the first image to be annotated is used for a segmentation task, mapping a mask of an object of interest in the first image to the first image before the specified size adjustment; A mapping result of the mask of the object of interest in the first image before the designated size adjustment is used as a labeling result of the object of interest in the first image.

[0057] In some exemplary embodiments, when the image annotation module 1440 annotates the object of interest in the first image based on the mask of the object of interest in the first image, it is specifically used to: When the first image to be annotated is used for an object detection task, mapping a mask of an object of interest in the first image to the first image before the specified size adjustment; Determine position information of a minimum circumscribed rectangle of an outline of the mask of the object of interest based on a mapping result of the mask of the object of interest in the first image before the specified size adjustment; The object of interest in the first image is marked based on the position information of the minimum circumscribed rectangle of the outline of the mask of the object of interest.

[0058] The image annotation device 1400 can realize Figure 1 to Figure 12 For details, please refer to Figure 1 to Figure 12 The image annotation method of the illustrated embodiment will not be described in detail.

[0059] Fig.15 The following is a schematic diagram of a computing device provided by an exemplary embodiment of the present application. Fig.15 As shown, the device includes: a memory 151 and a processor 152.

[0060] The memory 151 is used to store computer programs and can be configured to store various other data to support operations on the computing device. Examples of such data include instructions for any application or method operating on the computing device, contact data, phone book data, messages, pictures, videos, etc.

[0061] The processor 152 is coupled to the memory 151 and is used to execute the computer program in the memory 151 to: obtain initial bounding box information of the object of interest in the first image to be annotated, wherein the initial bounding box information is obtained based on the annotations of the annotator; Generate an image embedding vector based on the first image through an image encoder of an image segmentation model; Generate a sparse embedding vector based on the initial bounding box information through a hint encoder of an image segmentation model; A mask of the target of interest in the first image is generated based on the image embedding vector and the sparse embedding vector by a mask decoder of the image segmentation model, and the target of interest in the first image is labeled based on the mask of the target of interest in the first image.

[0062] By using the computing device provided in the embodiment of the present application, it is possible to obtain the initial bounding box information of the target of interest obtained by the annotator in the first image to be annotated, and generate an image embedding vector based on the first image through the image encoder of the image segmentation model, and generate a sparse embedding vector based on the initial bounding box information through the prompt encoder of the image segmentation model, and finally, by using the sparse embedding vector as the prompt information of the target of interest through the mask decoder of the image segmentation model, the mask of the target of interest in the first image is generated by searching in the image embedding vector, and finally, based on the mask of the target of interest in the first image, accurate annotation of the target of interest in the first image can be achieved. Since the image segmentation model can be used to predict a more accurate mask of the target of interest in the first image with the initial bounding box information as the prompt information, the annotation information finally determined based on the mask of the target of interest is more in line with the shape of the target of interest, thereby ensuring the accuracy of the annotation result, and at the same time, it can also reduce the reliability of the annotation result affected by the different experience and judgment criteria of the annotators.

[0063] Further, if Fig.15 As shown, the computing device also includes: a communication component 153, a display 154, a power component 155, an audio component 156 and other components. Fig.15 Only some components are shown schematically, and it does not mean that the computing device only includes Fig.15 In addition, depending on the different implementation forms of the traffic playback device, Fig.15 The components in the dashed box are optional components, not mandatory components. For example, when the computing device is implemented as a terminal device such as a smart phone, a tablet computer, or a desktop computer, it may include Fig.15 When the computing device is implemented as a server device such as a conventional server, cloud server, data center or server array, it may not include Fig.15 Components within the dashed box.

[0064] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0065] Accordingly, the present application also provides a computer program product, including a computer program / instruction, which, when executed, can implement the steps that can be performed by a computing device in the above-mentioned design processing method embodiment. Optionally, the computer program product, in addition to executing the above-mentioned application software testing method or the steps in the method embodiment.

[0066] Above Fig.15 The communication component in the device is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component may also include a near field communication (NFC) module, a radio frequency identification (RFID) technology, an infrared data association (IrDA) technology, an ultra-wideband (UWB) technology, a Bluetooth (BT) technology, etc.

[0067] Above Fig.15 The memory in the invention can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0068] Above Fig.15 The display in the embodiment includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0069] Above Fig.15 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0070] Above Fig.15 The audio component in the device may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0073] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0078] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0080] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image annotation method, characterized in that: include: Acquire initial bounding box information of an object of interest in a first image to be annotated, wherein the initial bounding box information is obtained based on annotations by an annotator; Generate an image embedding vector based on the first image through an image encoder of an image segmentation model; Generate a sparse embedding vector based on the initial bounding box information through a hint encoder of an image segmentation model; A mask of the target of interest in the first image is generated based on the image embedding vector and the sparse embedding vector by a mask decoder of the image segmentation model, and the target of interest in the first image is labeled based on the mask of the target of interest in the first image.

2. The method according to claim 1, characterized in that The obtaining of initial bounding box information of the object of interest in the first image to be labeled includes: When the first image to be annotated is used for a segmentation task, obtaining mask annotation information of an object of interest in the first image by an annotator; and generating a minimum circumscribed rectangular frame based on the mask annotation information as initial bounding box information of the object of interest in the first image; When the first image to be annotated is used for a target detection task, bounding box annotation information of the target of interest in the first image by the annotator is obtained as initial bounding box information of the target of interest in the first image.

3. The method according to claim 1, characterized in that The image encoder using the image segmentation model generates an image embedding vector based on the first image, including: Resizing the first image to a specified size to obtain a first image of the specified size; Obtaining, by an image encoder of an image segmentation model, a feature vector of the first image, and an absolute position code and a relative position code of the first image based on a first image of a specified size; The image encoder of the image segmentation model generates the image embedding vector based on the feature vector of the first image, and the absolute position encoding and relative position encoding of the first image.

4. The method according to claim 1, characterized in that The obtaining of initial bounding box information of the object of interest in the first image to be labeled includes: When the clarity of the first image to be marked is less than or equal to a preset clarity threshold, performing image enhancement on the first image so that the clarity of the first image after image enhancement is greater than the preset clarity threshold; Acquire initial bounding box information of the object of interest in the first image after the image enhancement.

5. The method according to claim 3, characterized in that The marking the object of interest in the first image based on the mask of the object of interest in the first image includes: When the first image to be annotated is used for a segmentation task, mapping a mask of an object of interest in the first image to the first image before the specified size adjustment; A mapping result of the mask of the object of interest in the first image before the designated size adjustment is used as a labeling result of the object of interest in the first image.

6. The method according to claim 3, characterized in that The marking the object of interest in the first image based on the mask of the object of interest in the first image includes: When the first image to be annotated is used for an object detection task, mapping a mask of an object of interest in the first image to the first image before the specified size adjustment; Determine position information of a minimum circumscribed rectangle of an outline of the mask of the object of interest based on a mapping result of the mask of the object of interest in the first image before the specified size adjustment; The object of interest in the first image is marked based on the position information of the minimum circumscribed rectangle of the outline of the mask of the object of interest.

7. An image annotation device, characterized in that: include: An information acquisition module, used to acquire initial bounding box information of an object of interest in a first image to be annotated, wherein the initial bounding box information is obtained based on annotations by annotators; A first generating module, configured to generate an image embedding vector based on the first image by using an image encoder of an image segmentation model; A second generation module is used to generate a sparse embedding vector based on the initial bounding box information through a hint encoder of the image segmentation model; An image annotation module is used to generate a mask of the object of interest in the first image based on the image embedding vector and the sparse embedding vector through a mask decoder of the image segmentation model, and to annotate the object of interest in the first image based on the mask of the object of interest in the first image.

8. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program to: Acquire initial bounding box information of an object of interest in a first image to be annotated, wherein the initial bounding box information is obtained based on annotations by an annotator; Generate an image embedding vector based on the first image through an image encoder of an image segmentation model; Generate a sparse embedding vector based on the initial bounding box information through a hint encoder of an image segmentation model; A mask of the target of interest in the first image is generated based on the image embedding vector and the sparse embedding vector by a mask decoder of the image segmentation model, and the target of interest in the first image is labeled based on the mask of the target of interest in the first image.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images