Image annotation method, device, electronic device and storage medium

Abstract

The present application relates to an image annotation method, device, electronic device and storage medium, wherein the image annotation method includes: by obtaining prompt words, annotating the image to be annotated according to the prompt words, unifying the target category corresponding to the prompt words; extracting the text features of the target category; obtaining the image to be annotated, extracting the image features of the image to be annotated; solidifying the text features of the target category to obtain text-strongly correlated features; performing feature enhancement on the image features of the image to be annotated to obtain image enhancement features; inputting the text-strongly correlated features and the image enhancement features into a preset training model for detection and segmentation to obtain an initial detection and segmentation result; performing accuracy verification on the initial detection and segmentation result to obtain a target detection and segmentation result, and saving the target detection and segmentation result to the target annotation file corresponding to the image to be annotated. Through the present application, the accuracy of image annotation is improved and the workload of manual review is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an image annotation method, apparatus, electronic device and storage medium. Background Art

[0002] In the field of deep learning vision, the improvement of algorithm model accuracy is highly dependent on the quality and size of the labeled data, especially with the advent of the era of large visual models, which rely on large-scale data sets for pre-training and fine-tuning. For example, the open source general object detection data set Objects365 contains 630,000 images, covering 365 categories, with up to 10 million bounding boxes; the open source ImageNet1k classification data set contains a total of 14,197,122 images, with a total of 1,000 categories. In image annotation work, the amount of annotation is huge, and the accuracy of annotation will also affect the quality of the data set.

[0003] Currently, image annotation can be divided into three methods: pure manual annotation, semi-automatic annotation, and fully automatic annotation. If pure manual annotation is used, when the amount of data is huge, it will consume a lot of manpower and affect the efficiency of algorithm development. Therefore, using the target detection algorithm to perform reasoning first to obtain the detection results, so as to achieve semi-automatic or automatic annotation has become the main research direction of image annotation. However, because the deep learning detection algorithm cannot guarantee that the detection results are 100% correct, usually after the algorithm automatically annotates, manual secondary annotation and review will be used to ensure the accuracy of the annotation.

[0004] There is currently no effective solution to the problems of low image annotation accuracy and large manual review workload in related technologies. Summary of the invention

[0005] In this embodiment, an image annotation method, device, electronic device and storage medium are provided to solve the problems of low image annotation accuracy and large manual review workload in related technologies.

[0006] In a first aspect, an image annotation method is provided in this embodiment, including:

[0007] Obtain prompt words, annotate the image to be annotated according to the prompt words, unify the target category corresponding to the prompt words; extract text features of the target category;

[0008] Acquire the image to be annotated, and extract the image features of the image to be annotated;

[0009] Solidify the text features of the target category to obtain text-strongly relevant features;

[0010] Perform feature enhancement on the image features of the image to be annotated to obtain image enhancement features;

[0011] Input text-strongly related features and image-enhanced features into the preset training model for detection and segmentation to obtain initial detection and segmentation results;

[0012] The accuracy of the initial detection and segmentation results is checked to obtain the target detection and segmentation results, and the target detection and segmentation results are saved in the target annotation file corresponding to the image to be annotated.

[0013] In some of the embodiments, the target categories corresponding to the unified prompt words include:

[0014] Translate the prompt words to obtain multiple translated prompt words;

[0015] The similarity of each translated prompt word is compared with all the categories in the preset prompt word library, and each translated prompt word is mapped to a category according to the principle of the highest similarity, and the category with the highest number of mappings is used as the target category of the prompt word.

[0016] In some of the embodiments, the image features of the image to be annotated are enhanced to obtain image enhancement features, including:

[0017] Divide the image to be annotated into several small image modules;

[0018] Extract image features of several small image modules as query image features;

[0019] Obtain a preset number of standard sample images, and extract image features of the standard sample images as reference image features; wherein the standard sample images are images that are correctly annotated in advance;

[0020] The query image features are aligned with the reference image features to obtain image enhancement features.

[0021] In some of the embodiments, obtaining a preset number of standard sample images and extracting image features of the standard sample images as reference image features includes:

[0022] Obtain a preset number of standard sample images;

[0023] Extract the image features of a preset number of standard sample images,

[0024] The image features of each standard sample image are fused to obtain the reference image features.

[0025] In some of the embodiments, feature alignment is performed on query image features and reference image features to obtain image enhancement features, including:

[0026] Performing correlation analysis on query image features and reference image features to obtain correlation analysis results;

[0027] The query image features whose correlation results are greater than a preset correlation threshold are used as image enhancement features.

[0028] In some of the embodiments, the initial detection and segmentation results include object boxes and object masks.

[0029] In some embodiments, the accuracy of the initial detection and segmentation results is checked to obtain the target detection and segmentation results, and the target detection and segmentation results are saved in the target annotation file corresponding to the image to be annotated, including:

[0030] Calculate the distance between the center point of the object frame and the center point of the object mask to obtain a first center point distance;

[0031] Obtain a preset number of standard sample images, and extract image features of the standard sample images as reference image features; wherein the standard sample images are images that are correctly annotated in advance;

[0032] Align the object frame with the reference image features to obtain the object frame image enhancement features;

[0033] Inputting the target category and the object frame image enhancement features into the preset training model for detection and segmentation to obtain a new detection and segmentation result; the new detection and segmentation result includes a new object frame and a new object mask;

[0034] Calculate the distance between the center point of the new object frame and the center point of the new object mask to obtain a second center point distance;

[0035] The difference between the first center point distance and the second center point distance is calculated. If the difference is less than the preset difference threshold, the detection and segmentation results corresponding to the new object box and the new object mask are used as the target detection and segmentation results, and the target detection and segmentation results are saved to the target annotation file corresponding to the image to be annotated.

[0036] In a second aspect, an image annotation device is provided in this embodiment, including: a first acquisition module, a second acquisition module, a feature enhancement module, a detection and segmentation module, and a verification module, wherein:

[0037] The first acquisition module is used to acquire prompt words, annotate the image to be annotated according to the prompt words, unify the target category corresponding to the prompt words, and extract text features of the target category;

[0038] The second acquisition module is used to acquire the image to be annotated and extract the image features of the image to be annotated;

[0039] The feature enhancement module is used to solidify the text features of the target category to obtain text-strongly related features; and to enhance the image features of the annotated image to obtain image enhancement features;

[0040] The detection and segmentation module is used to input text-related features and image enhancement features into a preset training model for detection and segmentation to obtain initial detection and segmentation results;

[0041] The verification module is used to verify the accuracy of the initial detection and segmentation results, obtain the target detection and segmentation results, and save the target detection and segmentation results to the target annotation file corresponding to the image to be annotated.

[0042] In a third aspect, an electronic device is provided in this embodiment, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the image annotation method described in the first aspect is implemented.

[0043] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the image annotation method described in the first aspect is implemented.

[0044] Compared with the related art, the image annotation method provided in this embodiment obtains prompt words, annotates the image to be annotated according to the prompt words, and unifies the target categories corresponding to the prompt words; extracts text features of the target category; obtains the image to be annotated, and extracts image features of the image to be annotated; solidifies the text features of the target category to obtain text-strongly correlated features; performs feature enhancement on the image features of the image to be annotated to obtain image enhancement features; inputs the text-strongly correlated features and the image enhancement features into a preset training model for detection and segmentation to obtain initial detection and segmentation results; performs accuracy verification on the initial detection and segmentation results to obtain target detection and segmentation results, and saves the target detection and segmentation results to the target annotation file corresponding to the image to be annotated, thereby solving the problems of low image annotation accuracy and large manual review workload, improving image annotation accuracy, and reducing manual review workload.

[0045] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 is a hardware structure block diagram of a terminal of the image annotation method of this embodiment;

[0048] Figure 2 is a flow chart of the image annotation method of this embodiment;

[0049] Figure 3 is a flow chart of image annotation result verification in this embodiment;

[0050] Figure 4 is a flow chart of another image annotation method of this embodiment;

[0051] Figure 5 FIG. 4 is a structural block diagram of the image annotation device of this embodiment. DETAILED DESCRIPTION

[0052] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0053] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the image annotation method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0055] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image annotation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0056] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0057] In this embodiment, an image annotation method is provided. Figure 2 is a flow chart of the image annotation method of this embodiment. Figure 2 As shown, the process includes the following steps:

[0058] Step S201, obtaining prompt words, annotating the image to be annotated according to the prompt words, unifying the target categories corresponding to the prompt words; and extracting text features of the target categories.

[0059] Specifically, image annotation refers to the process of adding labels or annotations to specific objects, scenes or concepts in an image. These labels usually exist in text form and describe the content or attributes in the image. In the prior art, semi-automatic annotation is mainly used. The semi-automatic annotation process is mainly divided into the following key steps: First, use an existing annotated data set or annotate a small batch of data sets in advance to train a deep learning network for different tasks (detection, segmentation); then use the trained network to infer the annotated data, obtain the inference result, and save the result to achieve automatic annotation; finally, the annotator re-annotates the automatic annotation results (for example, there are missed detections) or reviews (for example, there are detection errors that lead to annotation errors), thereby completing the end-to-end automatic annotation process.

[0060] However, in actual situations, firstly, due to the diversity of labeling tasks, the labeling accuracy through a single inference model is low, and it is often necessary to train different or even multiple inference models to cover different automatic labeling tasks. In view of the diversity of labeling tasks, the current method is mainly to train semantic segmentation networks and target detection networks, and then use integration and filtering algorithms to achieve deep learning semi-automatic image precision labeling methods. Or based on the labeled original data set and the map-enhanced data set, different models A, B, and C are trained to obtain their respective optimal models, and then automatic labeling is performed based on the trained models. Secondly, the visual big model is also used in automatic labeling. Using the visual big model, the image to be labeled is divided into several image embedding masks, and integrated to generate a model that can be displayed on the web page. The user hovers the mouse to generate a prompt area, and the user clicks the prompt area to establish the correct target block. However, this method requires manual intervention every time the labeling is performed, and it is highly dependent on manual review, which increases the workload of image labeling. Therefore, this embodiment realizes automatic image annotation through a training model for different annotation tasks. In this embodiment, the zero-sample detector GroundingDINO and the zero-sample segmenter SAM are mainly used in series to form the image annotation training model data large model Grounded-SAM in this embodiment. Using Grounding DINO as an open set object detector, any object in the image can be found through text description. Combined with the SAM model, a MASK can be generated for any object in the image or video, including objects and images that have not appeared in the training process. The combination of the two can enable the entire workflow to have the ability to detect everything, segment everything, and generate everything.

[0061] Specifically, firstly, the prompt word is obtained, and the target in the image to be annotated is annotated according to the prompt word, for example, all the pants in the image are annotated. At this time, the pants need to be translated into English first. However, when translating the Chinese word "pants" into English, multiple English words will be generated, such as: pants, kecks, stride, daksd, etc. Due to the different translations, the annotated targets may be inconsistent, thereby causing the problem of inaccurate image targets. For this reason, in this embodiment, the target categories corresponding to the prompt words are first unified, and all possible English translation words are uniformly mapped to one target category according to the similarity through a many-to-one mapping method, so as to obtain a unified target category, thereby improving the accuracy of image annotation. For example, the English text prompt word (prompt) annotated for the category "pants" in the training set by Grounding-DINO is trousers, because pants, kecks, stride, daksd and trousers in the training set have the highest similarity, so the prompt word of "pants" is uniformly mapped to trousers. Thereby, the prompt is solidified and becomes a limited category detection, avoiding errors caused by polysemous words and Chinese-English translation. Extract text features for the unified target categories. The extraction of text features requires different target prompts to be segmented by points. The target prompts can be descriptive text or straightforward categories. The text features of the target category are obtained by BERT (Bidirectional Encoder Representations from Transformers). The feature dimension of BERT output is 768, which is then mapped to 256 dimensions through an MLP layer to achieve unification with the image features.

[0062] Step S202: acquiring the image to be annotated and extracting image features of the image to be annotated.

[0063] Specifically, an image data set that needs to be labeled is obtained, wherein these images can be images of a specific scene that have been screened, such as frames extracted from a video, or unlabeled images in an existing data set. Secondly, the acquired image is preprocessed to improve the accuracy of subsequent feature extraction. Preprocessing mainly enhances the features of the image itself, including histogram equalization, contrast enhancement, and suppression of image background noise, highlighting the main structure, etc. Image features are extracted from the preprocessed image, wherein the extraction of image features can be based on visual features of the image, such as edges, contours, shapes, textures, and regions, etc.; it can also be based on statistical features, such as grayscale histogram features, moment features, etc.; or transformation coefficient features, such as Fourier transform, discrete cosine transform, wavelet transform, etc.; and algebraic features, such as singular values ​​of matrices, etc. In this embodiment, the specific features of the image can be selected according to actual conditions, and this embodiment does not make specific limitations on this.

[0064] Step S203, solidifying the text features of the target category to obtain text-strongly correlated features; and enhancing the image features of the image to be annotated to obtain image-enhanced features.

[0065] Specifically, the prior art directly annotates images based on the extracted text features and image features. However, due to the presence of a large number of irrelevant features, there is a certain interference in the annotation of the target object, thereby affecting the readiness of the target annotation. To this end, this embodiment first solidifies the text features of the target category and screens the image features of the image to be annotated. Among them, for the enhancement of text features, the n-gram feature extraction method can be used to solidify the text features of the target category to obtain text-strongly correlated features. When screening the image features of the image to be annotated, this embodiment mainly selects a small number of positive samples of the target category (i.e., correctly annotated image samples), aligns the features with the image features, and selects image features that are highly correlated with the positive samples as image enhancement features, thereby removing irrelevant features and reducing detection interference.

[0066] Step S204: input the text-strongly correlated features and the image-enhanced features into a preset training model for detection and segmentation to obtain an initial detection and segmentation result.

[0067] Specifically, the preset training model is a zero-sample detector GroundingDINO and a zero-sample segmenter SAM connected in series to form the image annotation training model data large model Grounded-SAM in this embodiment. The solidified text-strong correlation features and image enhancement features are input into Grounded-SAM, and the text-strong correlation features are used to guide the query selection of image enhancement features. Through language guidance, the model selects the image enhancement features that are most relevant to the text content, thereby aligning the text-strong correlation information with the image enhancement information. Since the zero-sample detector GroundingDINO and the zero-sample segmenter SAM are integrated in this embodiment, Grounding DINO can find any object in the image through text description, and the SAM model can generate MASK (mask) for any object in the image or video, and segment the sample, including objects and images that have not appeared in the training process. Therefore, detection and segmentation can be achieved simultaneously through a large model Grounded-SAM provided in this embodiment, thereby obtaining the initial detection and segmentation results.

[0068] Step S205 , performing accuracy verification on the initial detection and segmentation results to obtain target detection and segmentation results, and saving the target detection and segmentation results to a target annotation file corresponding to the image to be annotated.

[0069] Specifically, in order to ensure the accuracy of image annotation, this embodiment automatically verifies the accuracy of the initial detection and segmentation results. By guiding the target frame and the target mask of the initial detection and segmentation results to each other, the distance between the center point of the target frame and the center point of the target mask is first calculated to obtain a first center point distance, and then the target frame is added to a minimum number of positive samples, and the target frame is inferred again to obtain a new detection and segmentation result, that is, a new target frame and a new target mask. The distance between the center point of the new target frame and the center point of the new target mask is calculated to obtain a second center point distance, and the difference between the first center point distance and the second center point distance is calculated to obtain an adjustment amplitude of the initial detection and segmentation result. The size of the adjustment amplitude is automatically determined to verify whether the initial detection and segmentation result is accurate. If the adjustment amplitude does not meet the requirements, the newly obtained detection and segmentation result is used as a new verification object, and the new target frame is re-inferred, thereby performing verification again until the final adjustment amplitude is within a preset range. The automatic verification ends, the target detection and segmentation result is obtained, and the target detection and segmentation result is saved to a target annotation file corresponding to the image to be saved, wherein the target annotation file includes information such as the target category, frame coordinates, and segmentation mask.

[0070] Through the above steps S201 to S205, the prompt words are obtained, the image to be annotated is annotated according to the prompt words, and the target category corresponding to the prompt words is unified; the text features of the target category are extracted; the image to be annotated is obtained, and the image features of the image to be annotated are extracted; the text features of the target category are solidified to obtain text-strongly related features; the image features of the image to be annotated are enhanced to obtain image-enhanced features; the text-strongly related features and the image-enhanced features are input into the preset training model for detection and segmentation to obtain the initial detection and segmentation results. Compared with the current image annotation by combining deep learning detection algorithms and manual secondary annotation, this embodiment realizes multiple annotation tasks through a large visual model. First, the inference module in the large model is improved, and the target categories corresponding to the prompt words are unified through a many-to-one mapping method, so as to avoid the recognition results of the model being affected by the different translations of the prompt words; secondly, the text features of the target category are solidified, and the image features of the image to be annotated are enhanced with a small number of positive samples of the target category, so as to improve the performance of multimodal fusion and large models and improve the accuracy of image annotation; finally, the detection and segmentation results are automatically adjusted by calculating the distance between the center point of the target box and the center point of the target mask, so as to obtain accurate image annotation results, improve the accuracy of image standards, and reduce the workload of manual review.

[0071] In some of the embodiments, the target categories corresponding to the unified prompt words include:

[0072] The prompt word is translated to obtain multiple translated prompt words; each translated prompt word is compared with all categories in a preset prompt word library for similarity, and each translated prompt word is mapped to a category according to the principle of highest similarity, and the category with the highest number of mappings is used as the target category of the prompt word.

[0073] Specifically, the obtained prompt word is translated into Chinese and English. Due to the ambiguity of the meaning of the word, multiple translated English prompt words are generally obtained. For example, the Chinese prompt word is "pants", and the Chinese-English translation may obtain: pants, kecks, stride, daksd, etc. The translated prompt word is compared with all the categories preset in the Grounding-DINO training set for similarity. The similarity algorithm can use cosine similarity, Jaccard similarity or Euclidean distance, etc., which is not specifically limited in this embodiment. According to the similarity between each translated prompt word and each category in the training set, the category with the largest similarity is mapped to the translated prompt word, so that each translated prompt word corresponds to a category, and the number of categories obtained is counted, and the category with the highest mapping number is used as the target category of the prompt word. For example, pants, kecks, stride, daksd and trousers have the highest similarity, and they will all be mapped to trousers. Therefore, trousers is used as the target category of the prompt word "pants".

[0074] In another embodiment, the image features of the image to be annotated are enhanced to obtain image enhancement features, including:

[0075] The image to be annotated is segmented to obtain several small image modules; the image features of the several small image modules are extracted as query image features; a preset number of standard sample images are obtained, and the image features of the standard sample images are extracted as reference image features; wherein the standard sample images are images that are correctly annotated in advance; the query image features are aligned with the reference image features to obtain image enhancement features.

[0076] The step of obtaining a preset number of standard sample images and extracting image features of the standard sample images as reference image features includes:

[0077] A preset number of standard sample images are obtained; image features of each of the preset number of standard sample images are extracted, and the image features of each standard sample image are fused to obtain reference image features.

[0078] Among them, feature alignment is performed on the query image features and the reference image features to obtain image enhancement features, including:

[0079] Performing correlation analysis on query image features and reference image features to obtain correlation analysis results;

[0080] The query image features whose correlation analysis results are greater than a preset correlation threshold are used as image enhancement features.

[0081] Specifically, the image to be annotated is first segmented to obtain N×N small image modules (16×16 is used in this embodiment), and the feature extractor Image Backbone is used to extract image features from the 16×16 small image modules to obtain 16×16 image features, which are used as query image features;

[0082] Collect and organize n categories. The number of n categories can be determined according to the number of categories that need to be labeled. In this embodiment, multiple categories can be labeled at the same time. N positive samples are obtained for each category. The positive samples are standard sample images that have been labeled in advance. These positive samples can clearly show the category characteristics and have sufficient diversity to cover the variability of the category. For each positive sample image, image processing technology is used to identify and extract the minimum enclosing area of ​​each target object. The feature extractor Image Backbone is used to extract the image features of N positive samples in n categories, and the N image features in each category are feature fused, wherein the feature fusion can be achieved through a feature fusion network. For example, through the excellent processing ability of convolutional neural networks in the early stages of visual tasks and the efficiency in extracting long-distance features, feature fusion is achieved, and the image features after fusion of each category are used as reference image features.

[0083] The query image features and the reference image features are subjected to correlation analysis. The specific correlation analysis can be achieved by calculating the similarity between the features, such as using cosine similarity, correlation coefficient or other correlation measurement methods. A correlation threshold is preset in advance, and the calculated similarity is compared with the preset correlation threshold. The query image features with a similarity greater than the preset correlation threshold are used as image enhancement features, and the query image features with a similarity lower than the preset correlation threshold are discarded, that is, the image features with low correlation are discarded, thereby reducing detection interference items, achieving image feature enhancement, and improving the accuracy of subsequent image annotation.

[0084] In another embodiment, Figure 3 is a flow chart of image annotation result verification in this embodiment, such as Figure 3 As shown, the accuracy of the initial detection segmentation result is checked to obtain the target detection segmentation result, and the target detection segmentation result is saved to the target annotation file corresponding to the image to be annotated, including: the initial detection segmentation result includes an object frame and an object mask; the verification process includes:

[0085] Step S301, calculating the distance between the center point of the object frame and the center point of the object mask to obtain a first center point distance;

[0086] Specifically, the initial detection and segmentation results include object boxes and object masks. In this embodiment, the objects represent image annotation targets. These object boxes can accurately locate the position of the object, and the object mask provides more detailed target shape and contour information than the object box. If the image to be annotated is annotated with multiple categories at the same time, the distance between the center point of the object box and the center point of the object mask corresponding to each specific instance category output by the large model Grounded-SAM is calculated and recorded as the first center point distance. If there are n instance categories, it is counted as L 11 , L 12 ...L 1n .

[0087] Step S302, obtaining a preset number of standard sample images, and extracting image features of the standard sample images as reference image features; wherein the standard sample images are images that are correctly annotated in advance;

[0088] Specifically, in order to make the image annotation results more accurate, the same method as in the image feature enhancement method is used to collect and organize n categories. The number of n categories can be determined according to the number of categories that need to be annotated, and N positive samples are obtained for each category. For each positive sample image, image processing technology is used to identify and extract the minimum enclosing area of ​​each target object. The feature extractor Image Backbone is used to extract the image features of N positive samples in n categories, and the N image features in each category are fused to obtain the reference image features for the given category.

[0089] Step S303, aligning the object frame with the reference image features to obtain the object frame image enhancement features; inputting the target category and the object frame image enhancement features into the zero-sample detector for detection and segmentation to obtain a new detection and segmentation result; the new detection and segmentation result includes a new object frame and a new object mask; calculating the distance between the center point of the new object frame and the center point of the new object mask to obtain a second center point distance;

[0090] Specifically, in the same way as the above-mentioned image feature enhancement, the object frame is added to the reference image feature to obtain the image enhancement feature of the object frame, and then the target category and the image enhancement feature of the object frame are input into the large model Grounded-SAM for detection and segmentation to obtain a new detection and segmentation result, and then a new object frame and object mask are obtained, and then the distance between the center point of the new object frame and the center point of the new object mask is calculated to obtain the second midpoint distance. For example, L 21 , L 22 ...L 2n .

[0091] Step S304, calculating the difference ΔL between the first center point distance and the second center point distance n ;

[0092] Specifically, △L n =L 2n -L 1n ;

[0093] Step S305, determine the difference △L n Is it less than the preset difference threshold L? max , if yes, execute step S306; otherwise, execute step S303;

[0094] Step S306 : Use the detection and segmentation results corresponding to the new object frame and the new object mask as the target detection and segmentation results, and save the target detection and segmentation results to the target annotation file corresponding to the image to be annotated.

[0095] Through the above steps S301 to S306, by introducing the reference image features constructed by the positive sample, the object frame and the object mask can guide each other, and the functions of object frame fitting and mask pixel-level repair are realized, thereby achieving higher-precision annotation and automatic verification, reducing the workload of subsequent manual review and adjustment of annotation results.

[0096] This embodiment also provides an image annotation method. Figure 4 is a flow chart of another image annotation method of this embodiment. Figure 4 As shown, the process includes the following steps:

[0097] Step S401, obtaining prompt words, translating the prompt words, and obtaining a plurality of translated prompt words; comparing the similarity of each translated prompt word with all categories in a preset prompt word library, and mapping each translated prompt word to a category according to the principle of the highest similarity, and taking the category with the highest number of mappings as the target category of the prompt word;

[0098] Step S402, solidifying the text features of the target category to obtain text-strongly related features;

[0099] Step S403, obtaining the image to be annotated, and extracting image features of the image to be annotated;

[0100] Step S404, segmenting the image to be annotated to obtain a number of small image modules; extracting image features of the small image modules as query image features;

[0101] Step S405, obtaining a preset number of standard sample images, extracting image features of the standard sample images, and fusing the image features of each standard sample image to obtain reference image features; wherein the standard sample images are images that are correctly annotated in advance;

[0102] Step S406, performing correlation analysis on the query image feature and the reference image feature to obtain a correlation analysis result, and taking the query image feature whose correlation analysis result is greater than a preset correlation threshold as an image enhancement feature;

[0103] Step S407, inputting the text strong correlation features and the image enhancement features into a preset training model for detection and segmentation to obtain a current detection and segmentation result;

[0104] Step S408, calculating the distance between the center point of the object frame of the current detection and segmentation result and the center point of the object mask to obtain the current center point distance;

[0105] Step S409, adding the object frame of the current detection segmentation result to the reference image features in step S405, performing correlation analysis, and obtaining a new current detection segmentation result;

[0106] Step S410, calculating the distance between the center point of the new current detection segmentation result object frame and the center point of the new current object mask to obtain a new current center point distance;

[0107] Step S411, calculate the difference △L between the new current center point distance and the current center point distance n ;

[0108] Step S412, determine the difference △L n Is it less than the preset difference threshold L? max If yes, execute step S413; otherwise, execute step S414;

[0109] Step S413, taking the detection segmentation results corresponding to the new object frame and the new object mask as the target detection segmentation results, and saving the target detection segmentation results to the target annotation file corresponding to the image to be annotated;

[0110] Step S414: Use the new current detection segmentation result as the current detection segmentation result, use the new current center point distance as the current center point distance, and execute step S409.

[0111] Through the above steps S401 to S414, compared with the current image annotation by combining deep learning detection algorithms and manual secondary annotation, this embodiment realizes multiple annotation tasks through a large visual model. First, the reasoning module in the large model is improved, and the target categories corresponding to the prompt words are unified through a many-to-one mapping method, so as to avoid the recognition results of the model being affected by the different translations of the prompt words; secondly, a small number of positive samples of the target category are used to enhance the features of the image to be annotated and the text features of the target category, so as to improve the fusion of multimodal and the performance of the large model, and improve the accuracy of image annotation; finally, through the image annotation result verification module, the detection and segmentation results are automatically adjusted by calculating the distance between the center point of the target object frame and the center point of the object mask, and accurate image annotation results are obtained, the accuracy of the image standard is improved, and the workload of manual review is reduced.

[0112] In this embodiment, an image annotation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0113] Figure 5 is a structural block diagram of the image annotation device of this embodiment. Figure 5 As shown, the device 50 includes: a first acquisition module 51, a second acquisition module 52, a feature enhancement module 53, a detection and segmentation module 54 and a verification module 55, wherein:

[0114] The first acquisition module 51 is used to acquire prompt words, annotate the image to be annotated according to the prompt words, unify the target category corresponding to the prompt words, and extract text features of the target category;

[0115] The second acquisition module 52 is used to acquire the image to be annotated and extract image features of the image to be annotated;

[0116] The feature enhancement module 53 is used to solidify the text features of the target category to obtain text-strongly related features; and to enhance the image features of the image to be annotated to obtain image enhancement features;

[0117] The detection and segmentation module 54 is used to input the text strong correlation features and the image enhancement features into a preset training model for detection and segmentation to obtain an initial detection and segmentation result;

[0118] The verification module 55 is used to verify the accuracy of the initial detection and segmentation results, obtain the target detection and segmentation results, and save the target detection and segmentation results to the target annotation file corresponding to the image to be annotated.

[0119] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0120] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0121] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0122] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0123] S1, obtain prompt words, annotate the image to be annotated according to the prompt words, unify the target category corresponding to the prompt words; extract the text features of the target category;

[0124] S2, obtaining the image to be annotated and extracting image features of the image to be annotated;

[0125] S3, solidify the text features of the target category to obtain text-strongly related features;

[0126] S4, performing feature enhancement on the image features of the image to be annotated to obtain image enhancement features;

[0127] S5, inputting the text strong correlation features and the image enhancement features into the preset training model for detection and segmentation to obtain the initial detection and segmentation results;

[0128] S6, performing accuracy verification on the initial detection and segmentation results to obtain target detection and segmentation results, and saving the target detection and segmentation results to a target annotation file corresponding to the image to be annotated.

[0129] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0130] In addition, in combination with the image annotation method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the image annotation method. The storage medium stores a computer program, and when the computer program is executed by a processor, any one of the image annotation methods in the above embodiments is implemented.

[0131] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0133] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0134] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0136] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. An image annotation method, characterized in that: include: Obtain prompt words, annotate the image to be annotated according to the prompt words, unify the target categories corresponding to the prompt words; and extract text features of the target categories; Acquire the image to be annotated, and extract image features of the image to be annotated; Solidifying the text features of the target category to obtain text-strongly related features; Performing feature enhancement on the image features of the image to be annotated to obtain image enhancement features; Inputting the text strongly related features and the image enhancement features into a preset training model for detection and segmentation to obtain an initial detection and segmentation result; The accuracy of the initial detection and segmentation results is checked to obtain target detection and segmentation results, and the target detection and segmentation results are saved in a target annotation file corresponding to the image to be annotated.

2. The image annotation method according to claim 1, characterized in that: The target categories corresponding to the unified prompt words include: Translating the prompt word to obtain a plurality of translated prompt words; The similarity of each translated prompt word is compared with all categories in the preset prompt word library, and each translated prompt word is mapped to a category according to the principle of the highest similarity, and the category with the highest number of mappings is used as the target category of the prompt word.

3. The image annotation method according to claim 1, characterized in that: The step of enhancing the image features of the image to be annotated to obtain image enhancement features includes: Segmenting the image to be labeled to obtain a number of small image modules; Extracting image features of the plurality of small image modules as query image features; Obtain a preset number of standard sample images, and extract image features of the standard sample images as reference image features; wherein the standard sample images are images that are correctly annotated in advance; The query image feature and the reference image feature are aligned to obtain the image enhancement feature.

4. The image annotation method according to claim 3, characterized in that: The step of acquiring a preset number of standard sample images and extracting image features of the standard sample images as reference image features includes: Obtain a preset number of standard sample images; Extract the image features of a preset number of standard sample images, The image features of the standard sample images are fused to obtain reference image features.

5. The image annotation method according to claim 3, characterized in that: The step of aligning the query image feature with the reference image feature to obtain the image enhancement feature includes: Performing correlation analysis on the query image features and the reference image features to obtain a correlation analysis result; The query image feature whose correlation analysis result is greater than a preset correlation threshold is used as the image enhancement feature.

6. The image annotation method according to claim 1, characterized in that: The initial detection and segmentation result includes an object frame and an object mask.

7. The image annotation method according to claim 6, characterized in that: The accuracy check of the initial detection and segmentation result is performed to obtain a target detection and segmentation result, and the target detection and segmentation result is saved to a target annotation file corresponding to the image to be annotated, including: Calculating the distance between the center point of the object frame and the center point of the object mask to obtain a first center point distance; Obtain a preset number of standard sample images, and extract image features of the standard sample images as reference image features; wherein the standard sample images are images that are correctly annotated in advance; Aligning the object frame with the reference image features to obtain image enhancement features of the object frame; Inputting the target category and the object frame image enhancement features into the preset training model for detection and segmentation to obtain a new detection and segmentation result; the new detection and segmentation result includes a new object frame and a new object mask; Calculating the distance between the center point of the new object frame and the center point of the new object mask to obtain a second center point distance; Calculate the difference between the first center point distance and the second center point distance. If the difference is less than a preset difference threshold, use the detection and segmentation results corresponding to the new object box and the new object mask as the target detection and segmentation results, and save the target detection and segmentation results to the target annotation file corresponding to the image to be annotated.

8. An image annotation device, characterized in that: include: A first acquisition module, a second acquisition module, a feature enhancement module, a detection and segmentation module, and a verification module, wherein: The first acquisition module is used to acquire prompt words, annotate the image to be annotated according to the prompt words, unify the target category corresponding to the prompt words; and extract text features of the target category; The second acquisition module is used to acquire the image to be annotated and extract image features of the image to be annotated; The feature enhancement module is used to solidify the text features of the target category to obtain text-strongly related features; and to enhance the image features of the image to be annotated to obtain image enhancement features; The detection and segmentation module is used to input the text-strongly correlated features and the image enhancement features into a preset training model for detection and segmentation to obtain an initial detection and segmentation result; The verification module is used to perform accuracy verification on the initial detection and segmentation results to obtain target detection and segmentation results, and save the target detection and segmentation results to the target annotation file corresponding to the image to be annotated.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the image annotation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image annotation method according to any one of claims 1 to 7 are implemented.

Images