Image annotation method, device, equipment and medium

Abstract

The present application provides an image annotation method, apparatus, device and medium, which relate to the field of artificial intelligence technology, in particular to the field of machine learning, and are used to improve the efficiency of image annotation. The image annotation method determines the object similarity between the image to be annotated and the annotated image. When the similarity does not meet certain conditions, the initial category of the image to be annotated is determined based on the global features of the image to be annotated. Since more image information such as the object information and global features of the image to be annotated are combined, the accuracy of determining the initial category of the image to be annotated can be improved, thereby improving the accuracy of clustering the images to be annotated. In other words, the initial category corresponding to the clustered data set is more likely to be the true category of each image in the data set. This can reduce the subsequent re-judgment of the category of multiple images in the data set, thereby improving the efficiency and accuracy of manual annotation of the clustered data set.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence technology, and provides an image annotation method, apparatus, device, and medium. Background Art

[0002] With the continuous development of artificial intelligence technology, a variety of machine learning models have emerged. Most machine learning models require pre-training before they can be put into use. Supervised training is a commonly used pre-training method. The general principle of supervised training is to adjust the parameters of the machine learning model so that the model's output is closer to the labeled results of the samples. Therefore, this supervised training requires a large number of labeled samples.

[0003] Currently, samples are typically labeled using a combination of similarity aggregation and manual labeling. Specifically, samples with high object similarity are clustered and then each sample is manually labeled. However, this approach relies solely on clustering the objects in the sample, resulting in low clustering accuracy. This results in a large number of incorrectly clustered samples, reducing the efficiency of subsequent manual labeling. Summary of the Invention

[0004] The embodiments of the present application provide an image annotation method, apparatus, device, and medium for improving the efficiency of image annotation.

[0005] The present invention provides an image annotation method, including:

[0006] Determining similarities between the unlabeled object in the unlabeled image and the labeled objects in each labeled image, wherein each labeled image is labeled with a corresponding true category and each labeled image contains at least one labeled object;

[0007] If the maximum value among the obtained similarities is less than a first threshold, obtaining an initial category of the image to be labeled according to at least one global feature of the image to be labeled;

[0008] Clustering the images to be labeled into a dataset to be labeled corresponding to the initial categories;

[0009] In response to the labeling operation performed on the dataset to be labeled, a true category to which each to-be-labeled image in the dataset to be labeled belongs is determined.

[0010] The present invention provides an image annotation device, comprising:

[0011] a determination module, configured to determine the similarity between the to-be-annotated object in the to-be-annotated image and the annotated objects in each annotated image, wherein each annotated image is annotated with a corresponding true category and each annotated image contains at least one annotated object;

[0012] an obtaining module, configured to obtain an initial category of the image to be labeled based on at least one global feature of the image to be labeled if the maximum value among the obtained similarities is less than a first threshold;

[0013] A clustering module, configured to cluster the to-be-annotated images into to-be-annotated data sets corresponding to the initial categories;

[0014] The labeling module is configured to determine, in response to a labeling operation performed on the dataset to be labeled, a true category to which each to-be-labeled image in the dataset to be labeled belongs.

[0015] In a possible embodiment, the determining module is further configured to:

[0016] If the maximum value among the obtained similarities is not less than the first threshold, the category of the labeled image corresponding to the maximum value among the similarities is determined as the initial category of the image to be labeled.

[0017] In a possible embodiment, the obtaining module is further configured to:

[0018] Extracting at least one global feature of the image to be labeled;

[0019] Inputting the at least one global feature into a trained classification model to obtain a probability that the image to be labeled belongs to each of a plurality of categories, wherein the trained classification model is obtained by training based on the labeled images;

[0020] If the maximum value among the obtained multiple probabilities is not less than the second threshold, the category corresponding to the maximum value among the multiple probabilities is determined as the initial category of the image to be labeled.

[0021] In a possible embodiment, the marking module is specifically used to:

[0022] In response to a first selection operation triggered on the dataset to be annotated, displaying a plurality of groups included in the dataset to be annotated and a confirmation control associated with each group on an image annotation interface, wherein each group includes a plurality of images to be annotated;

[0023] In response to each confirmation operation triggered by the first confirmation control associated with a group among the multiple groups, a true category to which each to-be-annotated image in the group belongs is determined to be the initial category.

[0024] In a possible embodiment, the determining module is further configured to:

[0025] In response to the labeling operation performed on the dataset to be labeled, before determining the true category to which each to-be-labeled image in the dataset to be labeled belongs, displaying the dataset to be labeled and a plurality of candidate category labels associated with the dataset to be labeled in the image labeling interface, wherein the plurality of candidate category labels include a candidate category label corresponding to the initial category;

[0026] In response to a first selection operation on any candidate category label among the multiple candidate category labels, the initial category of the to-be-labeled data set is updated to the category corresponding to the selected candidate category label.

[0027] In a possible embodiment, the marking module is further configured to:

[0028] When the first confirmation control associated with each group is displayed on the image annotation interface, a first denial control associated with each group is displayed;

[0029] In response to a first deny operation triggered on a first deny control associated with a group among the plurality of groups, displaying a plurality of candidate category labels associated with the group;

[0030] In response to a second selection operation triggered by multiple candidate category labels associated with the group, the true category of each to-be-annotated image in the group is determined to be the category corresponding to the selected candidate category label.

[0031] In a possible embodiment, the marking module is further configured to:

[0032] When the image annotation interface displays a confirmation control associated with each group, a second denial control associated with each to-be-annotated image in each group is also displayed;

[0033] In response to a second deny operation triggered by a second deny control associated with an image to be labeled, displaying a plurality of candidate category labels for the image to be labeled;

[0034] In response to a third selection operation triggered by a plurality of candidate category labels associated with the image to be labeled, a true category of the image to be labeled is determined to be a category corresponding to the selected candidate category label.

[0035] In a possible embodiment, the true category is the parent category to which the image to be annotated belongs; and the annotation module is further configured to:

[0036] In response to a second selection operation triggered on the to-be-annotated dataset, displaying each to-be-annotated image and a corresponding plurality of sub-category labels in the image annotation interface, wherein the plurality of sub-category labels are a plurality of sub-category labels associated with the true category;

[0037] In response to each fourth selection operation triggered by a plurality of sub-category tags associated with an image to be annotated, the sub-category to which the image to be annotated belongs is determined to be the category corresponding to the selected sub-category tag.

[0038] An embodiment of the present application provides a computer device, including:

[0039] at least one processor, and

[0040] a memory communicatively coupled to the at least one processor;

[0041] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements any of the image annotation methods discussed above by executing the instructions stored in the memory.

[0042] An embodiment of the present application provides a computer storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer executes any of the image annotation methods discussed above.

[0043] Since the embodiments of the present application adopt the above technical solution, at least the following technical effects are achieved:

[0044] In the embodiment of the present application, the object similarity between the image to be annotated and each annotated image is judged, and the image to be annotated is clustered based on the judgment result. If the object similarity is high, the image to be annotated is directly clustered, and the image to be annotated can be clustered accurately and quickly. If the object similarity is low, the initial category of the image to be annotated is further determined based on the global features of the image to be annotated, and the image to be annotated is clustered based on the initial category. In this way, since the embodiment of the present application not only relies on object similarity to cluster images, but also combines the global features of the image to cluster, the clustering result is more accurate, thereby avoiding the situation of rethinking the category judgment of each image when manually annotating the clustered data set later, which is conducive to improving the efficiency of annotating the clustered data set. Moreover, since the true category of the image to be annotated in the clustered data set is more likely to belong to the initial category, the accuracy of image annotation can be relatively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1A Figure 1 is an application scenario applicable to an image annotation method provided in an embodiment of the present application;

[0046] Figure 1B Application scenarios applicable to an image annotation method provided in an embodiment of the present application Figure 2 ;

[0047] Figure 2 Flowchart 1 of an image annotation method provided in an embodiment of the present application;

[0048] Figure 3 An example diagram showing the similarity between an image to be annotated and an already annotated image provided in an embodiment of the present application;

[0049] Figure 4 An example diagram of an initial category of a confirmed data set provided in an embodiment of the present application;

[0050] Figure 5A The embodiment of this application provides Figure 4 Image annotation interface 1 displayed when annotating the dataset in;

[0051] Figure 5B The second image annotation interface displayed when annotating a group of a data set provided in an embodiment of the present application;

[0052] Figure 5C The third image annotation interface displayed when annotating an image to be annotated in a data set provided by an embodiment of the present application;

[0053] Figure 5D The fourth image annotation interface displayed when performing sub-category annotation on an image to be annotated in a data set provided by an embodiment of the present application;

[0054] Figure 5E The fifth image annotation interface displayed when performing subclass annotation on a group of a data set provided in an embodiment of the present application;

[0055] Figure 6 The process of the image annotation method provided in the embodiment of the present application Figure 2 ;

[0056] Figure 7 Provided in the embodiments of this application Figure 1B Schematic diagram of the interaction between the terminal and the server;

[0057] Figure 8 A schematic diagram of the structure of an image annotation device provided in an embodiment of the present application;

[0058] Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to better understand the technical solutions provided by the embodiments of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0060] In order to help those skilled in the art better understand the technical solution of this application, the terms involved in this application are introduced below.

[0061] 1. Artificial Intelligence (AI): It is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0062] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0063] 2. Computer Vision (CV): Computer vision is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying, tracking, and measuring targets. Further image processing is performed to transform computer-generated images into images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0064] 3. Machine Learning (ML): This is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0065] 4. Unstructured Data: Data in computer information systems is divided into structured and unstructured data. Unstructured data refers to data with irregular or incomplete structures, lacking a predefined data model, or data that is not easily represented using a two-dimensional logical table in a database. Unstructured data includes office documents in various formats, text, images, Extensible Markup Language (XML), Hypertext Markup Language (HTML), various reports, images, audio, and video.

[0066] 5. Object: In this application, it refers to the target object presented in the image. The image may include one or more target objects. When an image includes multiple target objects, these target objects may belong to different categories or the same category.

[0067] 6. Category: This is used to describe the type of the target object in the image. In the embodiment of the present application, the categories can be divided into multiple levels. For example, the target object can belong to Animal, which can be considered a parent category. The target object can specifically belong to Cat, which can be considered a subcategory under Animal.

[0068] 7. Candidate category labels: When annotating an image, the candidate category labels can be presented in the form of category names or colors. For example, if the category is animals, the corresponding label is "animal", or the animal category is represented by red. The category label used to represent the parent category can be called the parent category candidate category label, which can be further shortened to the parent category label. The category label used to represent the child category can be called the child category candidate category label, which can be further shortened to the child category label.

[0069] 8. Global features: Global features are used to describe the overall features of an image. Corresponding to global features, local features are used to describe the features of a local area of ​​an image. Global features include the shape, structure or deep features of an image. Deep features can also be called complex features. Shape features are used to describe the overall shape of an image. Shape features can be, for example, the quantified results of image contour information. Structural features can be, for example, the quantified results of geometric structures in an image. For example, the structural features of an image can be extracted through a graph convolutional neural network. Deep features can be the combined results of multiple features of the image to be annotated, or can be obtained by further processing one or more features in the image to be annotated, such as convolution processing or other deep learning processing. The relevant features of objects in the image involved in the embodiments of the present application can be understood as local features of the image.

[0070] In addition, the term "plurality" in the embodiments of the present application refers to two or more.

[0071] The following is a further explanation of the relevant technologies:

[0072] As AI commercialization accelerates, applications such as autonomous driving, facial recognition, and smart security are gaining momentum. Providing high-quality labeled data for training machine learning models has become a key component of the AI ​​industry's development.

[0073] Relevant data predicts that the amount of data generated by 2025 will reach 163ZB, of which 90% will be unstructured. This unstructured data can only realize its value through cleaning and annotation, which has created a continuous demand for cleaning and annotation. As a result, the data annotation industry has rapidly flourished and expanded.

[0074] In the related art, most of the methods cluster images based on object similarity and then annotate the clustered images. Specifically, the similarity between the objects in the image and the objects in the annotated images is determined, and the images are clustered into the category corresponding to the category of images with the highest similarity to the object. However, since the object similarity is determined only based on the features of the objects in the image, it is difficult for the object similarity results to accurately and comprehensively express the category of the image. In addition, the features of the objects in the image are affected by various factors, such as the clarity and brightness of the objects in the image, and these factors reduce the accuracy of the object similarity. Combined with the above analysis, the clustering results of the related art based on object similarity are not accurate, that is, there are a large number of images that are incorrectly classified in the clustered data set. Subsequently, users are required to reconsider the category for each image and perform manual annotation, which reduces the efficiency of image annotation.

[0075] In view of this, an embodiment of the present application provides an image annotation method, which utilizes the object similarity between the image to be annotated and the annotated image. If the object similarity is not less than a first threshold, the image to be annotated is directly clustered. Directly using object similarity for clustering can improve clustering efficiency; if clustering cannot be performed based on object similarity, the initial category corresponding to the image to be annotated can be determined based on the global features of the image to be annotated. Based on the determined initial category, the image to be annotated is clustered, and the data set obtained after clustering is annotated. In the embodiment of the present application, not only the object similarity between the image to be annotated and the annotated image is utilized, but also the global features of the image to be annotated are utilized when the object similarity judgment result does not meet the threshold. That is to say, the object features and global features of the image to be annotated are combined to classify the image to be annotated, thereby improving the accuracy of clustering the image to be annotated. Moreover, after clustering the images to be labeled, they are labeled. The true category of each image to be labeled in the clustered data set is more likely to belong to the initial category. Therefore, it is possible to avoid manually reconsidering and labeling a large number of images to be labeled. In addition, the clustered images to be labeled can be batch labeled, thereby improving the efficiency of image labeling.

[0076] Based on the above design ideas, the application scenarios of the image annotation method of the embodiment of the present application are introduced below.

[0077] Please refer to Figure 1A , which illustrates an application scenario applicable to the image annotation method provided in an embodiment of the present application, and includes a terminal 110. Terminal 110 corresponds to a user, who can manually upload an image to be annotated, and terminal 110 annotates the image. Terminal 110 can, for example, annotate the image using a pre-installed tool. The image annotation process involved is described below.

[0078] After annotating the image to be annotated, the corresponding annotation results can be obtained, and the annotation results can be used to train the machine learning model. Specifically, any type of machine learning model can be trained, such as various neural networks. This application does not limit the specific use of the annotation results.

[0079] Please refer to Figure 1B , which is another application scenario diagram applicable to the image annotation method provided in an embodiment of the present application, the application scenario diagram includes a terminal 110, a client 111 installed in the terminal 110, and a server 120 for supporting the functions of the client 111.

[0080] When a user needs to annotate an image, the user can upload the image to be annotated to the client 111. The client 111 sends the image to be annotated to the server 120, and the server 120 annotates the image. The processing process of the image to be annotated will be described below.

[0081] Figure 1B The client in the terminal 110 may be an application pre-installed in the terminal 110, a sub-application embedded in a third-party application, or a web page, etc. This application does not limit the specific form of the client 111. It should be noted that, Figure 1B Client 111 and Figure 1A The gadgets mentioned are different. Figure 1A The functions of the gadgets in the system do not need to rely on the background server to complete. Figure 1B The functions of the client 111 can be completed by relying on the background server.

[0082] Among them, the terminals mentioned above can be smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, gaming devices, smart TVs, smart bracelets, etc., but are not limited to these. The terminals and servers can be directly or indirectly connected via wired or wireless communications, and this application does not limit this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0083] It should be noted that the above application scenarios are examples of the image annotation method in the embodiment of the present application, but the image annotation method in the embodiment of the present application can actually be applied to any scenario of annotating images, and is not limited to Figure 1A and Figure 1B The application scenario shown.

[0084] Below Figure 1A The image annotation method of the embodiment of the present application is introduced by way of example using the terminal executing the image annotation method in FIG. Figure 2 , is a flow chart of an image annotation method provided in an embodiment of the present application, the method specifically comprising:

[0085] S201: The terminal determines the similarity between the object to be annotated in the image to be annotated and the annotated objects in each annotated image.

[0086] The terminal can obtain an image to be annotated in response to a user's image input operation, or the user can write an image to be annotated to a preset path, and the terminal can periodically or irregularly obtain the image to be annotated from the preset path. An image to be annotated can be understood as an image that needs to be annotated. Any image that needs to be annotated can be considered an image to be annotated. In the embodiment of the present application, one image to be annotated is used as an example to illustrate the image annotation method in the embodiment of the present application, but in actual practice, the terminal can use the same method to annotate multiple images to be annotated.

[0087] After obtaining the image to be annotated, the terminal can cluster the image to be annotated. Clustering can be understood as classifying the image to be annotated into a data set with the same category, so as to facilitate the subsequent unified annotation of multiple images to be annotated in the data set. In the embodiment of the present application, the terminal can use the similarity between the object to be annotated in the image to be annotated and the annotated object in the annotated image. For the sake of convenience of description, this similarity can also be called object similarity. The image to be annotated can be roughly clustered according to the various similarities obtained.

[0088] Specifically, when there is one object to be annotated in the image to be annotated, the terminal can determine the similarity between the object to be annotated and the annotated objects in the annotated image. When there are multiple objects to be annotated in the image to be annotated, the terminal can determine the similarity between each object to be annotated and the annotated objects in the annotated image. Alternatively, the terminal can determine an object to be annotated with significant features from the multiple objects to be annotated, and then determine the similarity between the object to be annotated with significant features and the annotated objects in each annotated image. A significant feature can be understood as a prominent feature of the object, such as a large proportion of pixels in the image occupied by the object, or a high degree of clarity of the object.

[0089] Similarly, there may be one or more annotated objects in the annotated image. When there is one annotated object in the annotated image, the similarity between the object to be annotated and the annotated object is directly determined; when there are multiple annotated objects in the annotated image, the similarity between the object to be annotated and each annotated object can be determined, or annotated objects with significant features can be screened out from the multiple annotated objects in the annotated image to determine the object to be annotated and the annotated objects with significant features.

[0090] This involves how to calculate the similarity specifically. For example, the terminal can calculate the similarity between the feature vectors of the object to be labeled and the labeled object, such as calculating the cosine similarity or Euclidean distance between the feature vectors. The feature vector can be, for example, the arrangement of pixel values ​​of the corresponding object, or the arrangement of color channel values ​​of the corresponding object, etc.

[0091] As an embodiment, the aforementioned annotated images may be pre-stored by the terminal or obtained from another device. An annotated image can be understood as an image in which at least one object has been annotated with a corresponding category. The terminal specifically establishes a sample library based on the multiple annotated images. The sample library includes the multiple annotated images and the categories of the annotated objects in each annotated image. The annotated images may belong to different categories, or they may belong to the same category.

[0092] Furthermore, the sample library may include many labeled images of each category. Therefore, in order to reduce the amount of similarity calculation, in an embodiment of the present application, the terminal can filter out a target labeled image from the multiple labeled images associated with each category. For example, the labeled image with the highest image quality among the multiple labeled images under each category can be used as the target labeled image. Image quality includes one or more of high image clarity, high image color saturation, and a large proportion of the pixels of the labeled objects in the image. After obtaining the target labeled images corresponding to each category in multiple categories, the terminal can directly determine the similarity between the objects to be labeled in the image to be labeled and the labeled objects in the target labeled images corresponding to each category.

[0093] In the embodiment of the present application, multiple labeled images can be screened, thereby reducing the amount of calculation in the similarity calculation process, and also avoiding the situation where the similarity calculation is inaccurate due to the low quality of the labeled images.

[0094] For example, see Figure 3 , is an example of the similarity results between the image to be labeled and each labeled image, a is the image to be labeled, b, c and d are labeled images under three different categories, b, c and d are marked with labeled objects, and the specific examples are as follows Figure 3 The terminal calculates the similarities between a and b, c and d, and determines that the similarity between a and b is 0.2, the similarity between a and c is 0.3, and the similarity between a and d is 0.4.

[0095] S202: If the maximum value of the obtained similarities is less than a first threshold, the terminal obtains an initial category of the image to be labeled according to at least one global feature of the image to be labeled.

[0096] After obtaining the various similarities, the terminal can determine the maximum value among the various similarities, that is, the similarity with the largest value among the various similarities, and then determine the relationship between the maximum value and a first threshold. The first threshold is a value set by the terminal and can be set to a large value to ensure accurate clustering of the images to be annotated.

[0097] Specifically, if the maximum value among the similarities is not less than the first threshold, then it means that the similarity between the image to be annotated and the annotated image corresponding to the maximum value is high, so the category corresponding to the annotated image with the maximum value in the similarity can be directly determined as the initial category of the image to be annotated. If the maximum value is less than the first threshold, then it means that the similarity between the image to be annotated and each annotated image is relatively low, which means that the possibility that the object to be annotated in the image to be annotated and each annotated object corresponding to the annotated image belong to the same category is low, or it is difficult to determine the category to which the object to be annotated in the image to be annotated belongs. Therefore, in an embodiment of the present application, the terminal can further extract at least one global feature of the image to be annotated, and determine the initial category of the image to be annotated based on at least one global feature. Global features, for example, include one or more of the shape, structure or deep features of the image to be annotated. The following describes how the terminal determines the initial category of the image to be annotated based on at least one global feature:

[0098] For example, the terminal can calculate the similarity between the global features of the image to be annotated and the global features of the annotated image, and then determine the category of the image to be annotated. This is different from the calculation of similarity in S201. In S201, the similarity between the object to be annotated and the annotated object is calculated, while in the embodiment of the present application, the similarity between the global features of the image to be annotated and the global features of the annotated image is calculated. When there are multiple global features in the image to be annotated, the similarity between each global feature in the image to be annotated and the corresponding global feature in the annotated image can be calculated, and the similarities corresponding to the multiple global features are weighted to obtain the similarity between the image to be annotated and the annotated image. Alternatively, the terminal can combine multiple global features into a vector or matrix, and then calculate the similarity between the combined results corresponding to the image to be annotated and the annotated image.

[0099] Alternatively, the terminal uses a classification model to determine the category of the image to be labeled based on at least one global feature. For example, the classification model can be used to fuse at least one global feature to output the category of the image to be labeled. The classification model can be trained using labeled images. For example, the labeled image is input into the classification model, and the model parameters of the classification model are adjusted according to the difference between the output of the classification model and the labeling result of the labeled image until the classification model converges to obtain a trained classification model. Since the classification model is actually a multi-classification model, the terminal can obtain the probability that the image to be labeled belongs to each category through the classification model, and can determine the initial category of the image to be labeled as the category corresponding to the maximum value among the probabilities.

[0100] There are many specific implementation forms of the classification model, such as neural networks, etc.

[0101] However, in some cases, the poor image quality of the image to be labeled may cause the probabilities of belonging to various categories output by the classification model to be similar. In this case, the obtained probabilities may not be of much reference value. Therefore, in an embodiment of the present application, the terminal can determine the probability that the image to be labeled belongs to each of the multiple categories, and determine that the maximum value of the multiple probabilities is not less than the second threshold value. In other words, it is determined that the probability of the image to be labeled belonging to a certain category is relatively high. At this time, the category corresponding to the maximum value of the multiple probabilities can be determined as the initial category of the image to be labeled, and then the image to be labeled can be clustered according to the initial category. If the maximum value of the multiple probabilities is less than the second threshold value, it means that the reference value of this classification result is not great, so the terminal can not cluster the image to be labeled.

[0102] Continue to use Figure 3 For example, if the terminal determines that the maximum value of the similarity between the image to be labeled and each labeled image is less than the first threshold, then the terminal can further extract the global features of the image to be labeled a, and then input the global features into the classification model based on the global features of the image to be labeled a to determine the initial category of the image to be labeled a.

[0103] It should be noted that in S201, the similarity is calculated based on the object to be annotated in the image to be annotated. Therefore, the initial category of the image to be annotated determined here can actually be understood as the initial category of the object to be annotated in the image to be annotated. If the image to be annotated includes multiple objects to be annotated, and if the multiple objects to be annotated belong to different categories, then the category of the image to be annotated determined by the terminal may include multiple categories.

[0104] S203: The terminal clusters the images to be labeled into a dataset to be labeled corresponding to the initial category.

[0105] After obtaining the initial category of the image to be labeled, the terminal can cluster the images to be labeled into a dataset to be labeled corresponding to the same initial category based on the initial category. Accordingly, the dataset to be labeled includes at least one image to be labeled and the initial category corresponding to each image to be labeled. If there are multiple images to be labeled belonging to different initial categories, then there will be multiple datasets to be labeled. In the embodiment of this application, the method of labeling the dataset to be labeled is introduced by taking the processing of one dataset to be labeled as an example.

[0106] As discussed above, if the image to be labeled corresponds to multiple initial categories, then the image to be labeled can be assigned to different data sets to be labeled. In the embodiment of the present application, the example of assigning the image to be labeled to one data set to be labeled is used for illustration. In practice, there is no limit on the number of data sets to be labeled to which the image to be labeled belongs.

[0107] S204 : The terminal determines the true category to which each to-be-annotated image in the to-be-annotated dataset belongs in response to the labeling operation performed on the to-be-annotated dataset.

[0108] In S203, the terminal has obtained the initial category of the images to be labeled in the dataset to be labeled, but the initial category is not necessarily the true category of the images to be labeled. Therefore, in the embodiment of the present application, after clustering the images to be labeled into the dataset to be labeled, the terminal can further determine the true category of the images to be labeled in the dataset to be labeled. For example, the user can manually label the dataset to be labeled. Specifically, for example, the user can label a batch of images to be labeled in the dataset to be labeled. In response to the labeling operation, the terminal determines the true category of each image to be labeled in the dataset to be labeled.

[0109] exist Figure 2 In the embodiment shown, the terminal can cluster the images to be annotated based on the similarity between the objects to be annotated in the images to be annotated and the annotated objects in the annotated images. Since the amount of computation involved in this process is not large, it is equivalent to first roughly clustering the images to be annotated using a simple similarity determination method. If the result of the similarity calculation does not meet certain conditions, the category of the images to be annotated is further determined based on the global features of the images to be annotated. That is to say, in the embodiment of the present application, the images can be clustered based on the local features of the images to be annotated, and then the images to be annotated are clustered in combination with the global features of the images to be annotated, so as to cluster the images to be annotated more accurately. In this way, it is more likely that each image to be annotated in the data set to be annotated obtained by clustering truly belongs to the initial category, which facilitates manual annotation of the data set to be annotated in the later stage, thereby improving the efficiency of annotating images.

[0110] Before executing S204, although the terminal obtains the initial category of the data set to be labeled, in order to ensure the accuracy of the initial category of the data set to be labeled, in an embodiment of the present application, the terminal can update the initial category of the data set to be labeled to the category corresponding to the selected candidate category label in response to the first selection operation performed on the candidate category label of the data set to be labeled.

[0111] In a specific implementation, the terminal can display the data set to be labeled and multiple candidate category labels associated with the data set to be labeled after clustering the images to be labeled, or in response to the user's third selection operation on the data set to be labeled, where the multiple candidate category labels include at least a candidate category label corresponding to the initial category.

[0112] As an embodiment, when displaying the data set to be labeled, each image to be labeled included in the data set to be labeled can be displayed separately, but the terminal may not be able to display all the images to be labeled in the data set to be labeled at one time. Therefore, the terminal can display some of the images to be labeled in the data set to be labeled, and display the overlapping results of other images to be labeled. The other images to be labeled include the images to be labeled in the data set to be labeled except for some of the images to be labeled.

[0113] Furthermore, when displaying the dataset to be labeled, the unique identifier of the dataset to be labeled can also be displayed in association, such as the name of the dataset to be labeled. Displaying the unique identifier of the dataset can facilitate users to distinguish different datasets.

[0114] For example, see Figure 4 , which is an example diagram of a dataset to be labeled displayed on the terminal, including multiple datasets, as shown in the following example: Figure 4 The terminal also displays multiple candidate category labels associated with each dataset, such as dataset 1, dataset 2, and dataset 3. Figure 4 There are many categories including street scenes, nature, art, people, animals, crowds, advertisements, pure text, comics and others.

[0115] After displaying the dataset to be labeled and the associated multiple candidate category labels, the user can select any candidate category label from the multiple candidate category labels for the dataset to be labeled. For example, clicking on any candidate category label is equivalent to performing a first selection operation. In response to the first selection operation, the terminal determines that the initial category of the dataset to be labeled is the category corresponding to the currently selected candidate category label, thereby updating the initial category of the dataset to be labeled to the category corresponding to the selected candidate category label. Of course, the user can also select a candidate category label corresponding to the initial category. In this case, the updated initial category of the dataset to be labeled is actually the same as the initial category before the update.

[0116] To simplify user operations, in an embodiment of the present application, when the terminal displays multiple candidate category labels associated with a dataset to be labeled, the candidate category label corresponding to the previously determined initial category can be highlighted, making it easier for the user to determine the initial category of the dataset to be labeled. There are various ways to highlight the candidate category label, such as bolding, highlighting, or marking the candidate category label with a different color.

[0117] For example, continue to refer to Figure 4 In the example shown, the user clicks on the candidate category label associated with dataset 1 using the mouse. Figure 4 After clicking the street scene in the image, the terminal responds to the click operation, thereby determining that the initial category of the dataset to be labeled is street scene.

[0118] In an embodiment of the present application, the terminal may support manual updating of the initial category of the dataset to be labeled, further ensuring the accuracy of the initial category of the dataset to be labeled, and facilitating improving the efficiency of subsequent labeling of the dataset to be labeled.

[0119] As an embodiment, after obtaining the initial category of the data set to be labeled, there may be multiple data sets to be labeled with the same initial category. Therefore, the terminal can merge the data sets to be labeled with the same initial category, which can reduce the number of data sets and facilitate subsequent batch labeling of the data sets.

[0120] The following is a specific example of the method of determining the true category of each image to be labeled in the dataset to be labeled involved in S204:

[0121] After obtaining the datasets to be annotated, the user can select one of them. In response to the first selection operation triggered for the dataset to be annotated, the terminal displays multiple groups corresponding to the dataset to be annotated and a confirmation control associated with each group on the image annotation interface. Each group includes multiple images to be annotated. The image annotation interface is used to annotate images.

[0122] As an embodiment, in order to facilitate users to match groups with corresponding confirmation controls, in an embodiment of the present application, the terminal can divide the image annotation interface into an annotation sample area and an operation area. The annotation sample area is used to display the images to be annotated associated with each group, and the operation area is used to display the confirmation controls associated with each group.

[0123] As an embodiment, to facilitate display or facilitate user annotation of a group, the terminal may divide the dataset to be annotated into a plurality of groups according to a fixed number, and arrange the groups along a first direction of the image annotation interface. The first direction may be, for example, a direction from top to bottom along the image annotation interface, where the "top" here is referenced to a horizontal plane.

[0124] For example, see Figure 5A , which is an example diagram of multiple groups in a dataset to be labeled displayed on a terminal. The dataset to be labeled displays multiple groups 501, each group 501 includes four images to be labeled, and each group is associated with a confirmation control 502. When the user clicks Figure 5A After the confirmation control 502 associated with the first group on the left is shown, it is equivalent to performing a confirmation operation. Therefore, the terminal responds to the confirmation operation and further determines that the true category of each to-be-annotated image in the group is the initial category.

[0125] The terminal can display the confirmation control of each group in multiple groups, but in actual operation, since the actual categories of multiple groups are not necessarily the initial categories, the user may not necessarily operate the confirmation control of each group in the multiple groups. Therefore, the terminal actually responds separately according to the user's operation. The following example introduces the process in which the user confirms the confirmation control of a group, and the terminal responds to the confirmation operation to determine the actual category to which the group belongs.

[0126] If the user determines that the true category of each to-be-annotated image in a group belongs to the initial category, then the user can click the confirmation control associated with the group, which is equivalent to performing a confirmation operation. The terminal can respond to the confirmation operation triggered by the confirmation control associated with the group and determine that the true category of each to-be-annotated image in the group belongs to the initial category. The group here can be any one of multiple groups, and this application does not limit this.

[0127] In an embodiment of the present application, the terminal can batch-label the dataset to be labeled in groups, that is, the true category to which each image to be labeled in a group belongs can be determined at one time. Since the clustering accuracy is high in the embodiment of the present application, it is more likely to perform batch labeling in groups, which simplifies the process of manually labeling images and improves the efficiency of image labeling.

[0128] Correspondingly, the terminal may further display a first denial control associated with each group, and the user may perform a first denial operation on the first denial control associated with each group. Of course, in actual application, the user may perform the first denial operation on some or all of the multiple groups, and the terminal may respond according to the user's specific operation. The following example takes the user performing the first denial operation on a group as an example for description:

[0129] A user can click on the first deny control associated with a group, which is equivalent to performing a first deny operation. The terminal responds to the first deny operation and determines that the true category of the group is not the initial category. To facilitate the user to further label the true category of the group, the terminal can also display multiple candidate category labels associated with the group after responding to the first deny operation. The user can select a candidate category label from the multiple candidate category labels, which is equivalent to performing a second selection operation. The terminal responds to the second selection operation and determines that the true category of the group is the category corresponding to the selected candidate category label.

[0130] For example, continue to use Figure 5A For example, when the user clicks Figure 5A The terminal responds to the click operation on the first deny control 503, thereby displaying the following: Figure 5B As shown in the multiple candidate category tags 504, when the user selects comics from the multiple candidate category tags 504, the terminal determines that the true category of the group is comics based on the user's selection operation.

[0131] In an embodiment of the present application, the terminal can re-determine the true category of the group for the data set to be labeled in groups, so as to quickly and accurately determine the true category of the corresponding group and improve the labeling efficiency.

[0132] Furthermore, in addition to supporting the processing of the dataset to be labeled in groups, the terminal can also support the processing of individual images to be labeled to meet the processing requirements of individual images to be processed.

[0133] Specifically, when displaying multiple groups, the terminal may display, in addition to the confirmation control and first rejection control associated with each group, a second rejection control associated with each image to be annotated in each group. The user can annotate each image to be annotated individually, and the terminal will respond based on the user's specific operation. The following example illustrates the terminal's response to a user's operation on a specific image to be annotated:

[0134] When the user determines that the true category of a certain image to be labeled is not the initial category, the user can perform a second denial operation on the second denial control, such as triggering the second denial control, and the terminal can determine that the true category of the image to be labeled is not the initial category. Furthermore, the terminal can display multiple candidate category labels of the image to be labeled for the user to select. The user can select one of the multiple candidate category labels, which is equivalent to performing a third selection operation. The terminal can respond to the third selection operation performed on the multiple candidate category labels of the image to be labeled, and then the terminal can determine the category corresponding to the selected candidate category label as the true category of the image to be labeled.

[0135] For example, continue to use Figure 5A In the example shown, the user clicks the second deny control 506 of a certain image to be annotated in a group, which is equivalent to performing a second deny operation, so the terminal can display the following Figure 5C As shown in the multiple candidate category tags 507, when the user clicks on an animal among the multiple candidate category tags, the category of the image to be labeled is determined to be an animal.

[0136] In the embodiment of the present application, a single image to be annotated can be annotated by category, which facilitates the user to annotate a single image to be annotated and improves annotation efficiency. In addition, in the embodiment of the present application, the dataset to be annotated can be annotated in groups, or a single image to be annotated can be annotated, which can improve the flexibility of image annotation and facilitate image annotation for users.

[0137] As an embodiment, in addition to displaying a confirmation control and a first denial control associated with each group, a second confirmation control associated with each to-be-annotated image in each group may also be displayed. In response to a click operation on the second confirmation control, the terminal performs a second confirmation operation, and determines that the true category of the to-be-annotated image is the initial category.

[0138] For example, continue to refer to Figure 5C , when the user targets Figure 5C Clicking the second confirmation control 505 in the dialog box is equivalent to performing a second confirmation operation, so the terminal determines the true category of the image to be labeled as the initial category.

[0139] When annotating the image to be annotated, the image to be annotated may be annotated at multiple levels to obtain a more detailed category of the image to be annotated. Therefore, in an embodiment of the present application, the terminal may support sub-category annotation of the dataset to be annotated.

[0140] Specifically, after determining the true category of each image in the dataset, the terminal can cluster the images belonging to the same true category to obtain a clustered dataset, and then perform sub-category labeling on the clustered dataset. Alternatively, the terminal can directly perform sub-category labeling on the dataset belonging to the same initial category as described above.

[0141] When the user needs to perform subcategory annotation on a certain dataset to be annotated, he can click or select a certain dataset to be annotated, which is equivalent to performing a second selection operation. The terminal can respond to triggering the second selection operation for the dataset to be annotated and display each image to be annotated in the image annotation interface, as well as multiple sub-category labels associated with each image to be annotated.

[0142] The user can select multiple sub-category tags associated with the image to be annotated, triggering the fourth selection operation, and then determine that the sub-category of the image to be annotated is the category corresponding to the sub-category tag selected by the user. This involves how the terminal determines which sub-category tags to display, as illustrated by the following example:

[0143] For example, the terminal can pre-store the subcategories under each parent category, and then display the subcategory labels associated with each parent category. In this way, the terminal does not need to filter the displayed content, simplifying the terminal's processing. Alternatively, the terminal can store each parent category and the subcategories associated with the parent category in a hierarchical relationship. After the terminal determines the dataset to be labeled selected by the user, it determines the subcategories associated with the parent category based on the parent category to which the image to be labeled in the dataset to be labeled belongs, and then displays the subcategory labels corresponding to each subcategory. This allows for targeted display, making it easier for users to select subcategory labels.

[0144] For example, continue to use Figure 5C In the example shown, when the terminal determines that the parent category of the dataset to be labeled is animal, and determines that the user needs to label the dataset to be labeled with a sub-category, the following is displayed: Figure 5D The image annotation interface shown in FIG, which displays each image to be annotated in the dataset to be annotated, and multiple sub-category labels associated with each image to be annotated, Figure 5D Taking the fourth picture on the left as an example, the terminal determines that the parent category of the image to be labeled is plant, and then displays multiple sub-category labels 508 associated with the plant, as shown in FIG. Figure 5D "Sunflower", "Cactus" and "Begonia" are shown in the picture.

[0145] After displaying the image to be annotated and the sub-category label associated with the image to be annotated, the user can click on the corresponding sub-category label, which is equivalent to performing a fourth selection operation. The terminal responds to the fourth selection operation performed on the image to be annotated, thereby determining the sub-category of the image to be annotated.

[0146] For example, continue to refer to Figure 5D When the user clicks Figure 5D When the user selects "Cactus" in the selection list, the fourth selection operation is performed. The terminal responds to the fourth selection operation and determines Figure 5D The subcategory of the fourth image is Cactus.

[0147] In an embodiment of the present application, a plurality of sub-category labels can be displayed for selection based on the parent category of the image to be annotated for the user to select. The terminal can determine the sub-category of the image to be annotated based on the user's selection operation, thereby achieving more detailed annotation of the image to be annotated.

[0148] As an embodiment, when displaying the images to be annotated in the image annotation interface, the images to be annotated can also be displayed in groups. At the same time, multiple sub-category labels associated with each group can also be displayed. The method of displaying the multiple sub-category labels associated with each group can refer to the method of displaying the multiple sub-category labels associated with the images to be annotated discussed above, and will not be repeated here.

[0149] Similarly, the terminal can respond to the fifth selection operation of the sub-category label for a group and further determine the sub-category label associated with each image to be annotated in the group. In this embodiment, the user can batch confirm the sub-category label corresponding to each group, improving the efficiency of image annotation.

[0150] In the embodiment of the present application, the sub-category under the parent category is taken as an example, but in fact the sub-category can be further subdivided into the next level of sub-category. The terminal can mark the next level of sub-category for the marked image in the same way as marking the sub-category. The specific marking method can refer to the content discussed above and will not be repeated here.

[0151] For example, continue to use Figure 5C For example, the terminal displays the following information based on the user's selected operation: Figure 5E The image annotation interface shown in the figure displays the images to be annotated in groups, and each group is associated with multiple corresponding sub-category labels 509 for user selection. When the user selects any one of the multiple sub-category labels 509, the sub-categories of all the images to be annotated under the group are determined.

[0152] As an embodiment, as discussed above, there may be images to be labeled that cannot be clustered. For example, the maximum value of the similarity between the image to be labeled and each image to be labeled is less than a first threshold, and the category of the image to be labeled cannot be determined based on the image to be labeled. In this case, the terminal can cluster these images to be labeled that cannot be clustered, and subsequent users can label these images to be labeled that cannot be clustered individually.

[0153] After determining the true category of each image to be labeled, each image to be labeled can be associated with the corresponding true category, or the true category can be marked on the image to be labeled in a specific form, thereby completing the labeling of the image to be labeled. After obtaining the labeled image, the labeled image can be used to optimize the classification model in the previous article to obtain a more accurate classification model.

[0154] As an embodiment, after obtaining the images to be annotated, these images to be annotated may be stored in a sample library, so as to continuously expand the amount of data in the sample library.

[0155] In order to more clearly illustrate the image annotation method in the embodiment of the present application, the following takes the image annotation method executed by the terminal as an example. Figure 6 The image annotation flowchart shown is introduced:

[0156] S601: Obtain an image to be labeled.

[0157] The method for the terminal to obtain the image to be annotated can refer to the content discussed above and will not be repeated here.

[0158] S602: Perform similarity judgment on the image to be annotated.

[0159] The terminal can determine the similarity between the image to be annotated and each annotated image in the sample library, and then determine whether the maximum value of each similarity is greater than a first threshold. If it is not less than the first threshold, then execute S603, that is, determine the initial category of the image to be annotated, specifically determine the category of the annotated image corresponding to the maximum value as the initial category of the image to be annotated. If the maximum value is less than the first threshold, then execute S604, that is, extract features of the image to be annotated, specifically extracting the shape, structure, or deep features of the image to be annotated.

[0160] As an embodiment, the first threshold may be a high threshold. The high threshold may be understood as the value of the first threshold being greater than a preset value, for example, the value may be 0.9.

[0161] S605, output the result.

[0162] Based on the features of the image to be labeled, the terminal executes S603, i.e., determines the initial category of the image to be labeled based on the features of the image to be labeled. Specifically, these features can be input into a neural network to obtain the initial category of the image to be labeled. If the initial category of the image to be labeled cannot be determined based on the features of the image to be labeled, the terminal can execute S606, i.e., treat the image to be labeled as a single sample. The terminal can further execute S607, i.e., label the single sample based on the manual labeling operation performed by the user.

[0163] S608, cluster labeling.

[0164] After obtaining the initial categories of the images to be labeled, the terminal may cluster the images to be labeled into corresponding data sets, and determine the initial categories corresponding to the data sets according to the selection operation of the initial categories of the data sets.

[0165] S609: Obtain the annotation results.

[0166] The terminal obtains the annotation results of the data set according to the corresponding annotation operations. The annotation operations involved and the method of obtaining the annotation results of the data set can be referred to the content discussed above and will not be repeated here.

[0167] Furthermore, the terminal can optimize the neural network based on the labeling results and store the labeling results in a sample library to obtain an optimized sample library.

[0168] In an embodiment of the present application, the images to be annotated are first preliminarily clustered based on the similarity between the images to be annotated and the annotated images in the sample library. If the similarity of the images to be annotated does not meet the conditions, the initial category of the images to be annotated is further determined based on the features of the images to be annotated. In this way, the images to be annotated can be clustered as accurately as possible. Since the clustering results are more accurate, the efficiency of subsequent data set annotation can be improved.

[0169] Based on the above Figure 1B The application scenario shown takes the server and terminal collaboratively completing the image annotation method as an example to introduce the image annotation method involved in the embodiment of the present application:

[0170] Please refer to Figure 7 , is a diagram of the interaction process between the server and the terminal, which includes:

[0171] S701: The terminal obtains an image to be annotated.

[0172] The method of obtaining the image to be annotated can refer to

[0173] S702: The terminal sends the image to be annotated to the server.

[0174] S703: The server determines the similarity between the object to be annotated in the image to be annotated and the annotated objects in each annotated image.

[0175] The manner in which the server determines the similarity between the image to be annotated and each annotated image can refer to the content discussed above and will not be repeated here.

[0176] S704: The server obtains an initial category of the image to be labeled based on at least one global feature of the image to be labeled.

[0177] The specific method for the server to determine the initial category of the image to be labeled can be referred to the content discussed above and will not be repeated here.

[0178] S705: The server clusters the images to be labeled into a dataset to be labeled corresponding to the initial category.

[0179] The specific method of clustering the images to be labeled by the server can refer to the content discussed above and will not be repeated here.

[0180] S706: The server sends the dataset to be labeled to the terminal.

[0181] S707: The terminal sends the marking operation information to the server.

[0182] After obtaining the dataset to be annotated, the terminal can display the dataset to be annotated and obtain annotation operation information in response to the annotation operation performed on the dataset to be annotated. The annotation operation information is used to describe the annotation operation. The specific form of the annotation operation can be referred to the content discussed above and will not be repeated here.

[0183] S708: The server determines the true category of each image to be labeled in the dataset to be labeled according to the labeling operation information.

[0184] S709: The server feeds back the annotation results to the terminal.

[0185] The labeling result includes the image to be labeled and the true category to which the image to be labeled belongs.

[0186] In addition, the server can also execute any of the image annotation methods discussed above, which will not be repeated here.

[0187] In the embodiment of the present application, the terminal and the server collaborate to annotate images, which can relatively reduce the processing load of the terminal. In addition, in the embodiment of the present application, the images to be annotated can be effectively clustered. On the one hand, it can quickly complete the screening of images for annotating when a specific data set is required. On the other hand, it can effectively reduce the repeated judgment operations on the images to be annotated during manual annotation, and the use of batch operations can improve the efficiency of manual annotation.

[0188] Based on the same inventive concept, the embodiment of the present application provides an image annotation device, which can be used to implement the functions of the terminal or server mentioned above. Figure 8 , the device comprises:

[0189] Determination module 801, for determining similarities between the unlabeled object in the unlabeled image and the labeled objects in each labeled image; wherein each labeled image is labeled with a corresponding true category, and each labeled image contains at least one labeled object;

[0190] an obtaining module 802 for obtaining an initial category of the image to be labeled based on at least one global feature of the image to be labeled if the maximum value among the obtained similarities is less than a first threshold;

[0191] Clustering module 803, used to cluster the images to be labeled into the dataset to be labeled corresponding to the initial category;

[0192] The labeling module 804 is configured to determine the true category to which each to-be-labeled image in the to-be-labeled dataset belongs in response to the labeling operation performed on the to-be-labeled dataset.

[0193] In a possible embodiment, the determining module 801 is further configured to:

[0194] If the maximum value among the obtained similarities is not less than the first threshold, the category of the labeled image corresponding to the maximum value among the similarities is determined as the initial category of the image to be labeled.

[0195] In a possible embodiment, the obtaining module 802 is further configured to:

[0196] Extracting at least one global feature of the image to be annotated;

[0197] Inputting at least one global feature into a trained classification model to obtain a probability that the image to be labeled belongs to each of a plurality of categories, wherein the trained classification model is obtained by training based on the labeled images;

[0198] If the maximum value among the obtained multiple probabilities is not less than the second threshold, the category corresponding to the maximum value among the multiple probabilities is determined as the initial category of the image to be labeled.

[0199] In a possible embodiment, the marking module 804 is specifically configured to:

[0200] In response to a first selection operation triggered on the dataset to be annotated, displaying a plurality of groups included in the dataset to be annotated and a confirmation control associated with each group on the image annotation interface, wherein each group includes a plurality of images to be annotated;

[0201] In response to each confirmation operation triggered by the first confirmation control associated with a group among the multiple groups, a true category to which each to-be-annotated image in the group belongs is determined as an initial category.

[0202] In a possible embodiment, the determining module 801 is further configured to:

[0203] In response to a labeling operation performed on the dataset to be labeled, before determining the true category to which each to-be-labeled image in the dataset to be labeled belongs, displaying the dataset to be labeled and a plurality of candidate category labels associated with the dataset to be labeled in an image labeling interface, wherein the plurality of candidate category labels includes a candidate category label corresponding to the initial category;

[0204] In response to a first selection operation on any candidate category label among the plurality of candidate category labels, the initial category of the to-be-labeled data set is updated to the category corresponding to the selected candidate category label.

[0205] In a possible embodiment, the marking module 804 is further configured to:

[0206] When the first confirmation control associated with each group is displayed on the image annotation interface, a first denial control associated with each group is displayed;

[0207] In response to a first negation operation triggered on a first negation control associated with a group among the plurality of groups, displaying a plurality of candidate category labels associated with the group;

[0208] In response to a second selection operation triggered by multiple candidate category labels associated with a group, the true category of each to-be-annotated image in the group is determined to be the category corresponding to the selected candidate category label.

[0209] In a possible embodiment, the marking module 804 is further configured to:

[0210] When the image annotation interface displays a confirmation control associated with each group, a second denial control associated with each to-be-annotated image in each group is also displayed;

[0211] In response to a second deny operation triggered by a second deny control associated with an image to be labeled, displaying a plurality of candidate category labels for the image to be labeled;

[0212] In response to a third selection operation triggered by a plurality of candidate category labels associated with an image to be labeled, a true category of the image to be labeled is determined to be a category corresponding to the selected candidate category label.

[0213] In a possible embodiment, the true category is the parent category to which the image to be annotated belongs; the annotation module 804 is further configured to:

[0214] In response to a second selection operation triggered on the dataset to be annotated, displaying each image to be annotated and a corresponding plurality of sub-category labels in the image annotation interface, wherein the plurality of sub-category labels are a plurality of sub-category labels associated with the true category;

[0215] In response to each fourth selection operation triggered by a plurality of sub-category tags associated with an image to be annotated, the sub-category to which the image to be annotated belongs is determined to be the category corresponding to the selected sub-category tag.

[0216] It should be noted that Figure 8 The device in can also be used to implement any of the image annotation methods discussed above, which will not be described in detail here.

[0217] Based on the same inventive concept, the present application embodiment provides a computer device, please refer to Figure 9 , the computer device includes a processor 901 and a memory 902.

[0218] The processor 901 may be a central processing unit (CPU) or a digital processing unit, etc. The specific connection medium between the memory 902 and the processor 901 is not limited in the embodiment of the present application. Figure 9 The memory 902 and the processor 901 are connected via a bus 903. Figure 9 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus 903 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0219] The memory 902 may be a volatile memory, such as a random-access memory (RAM); a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 902 may be a combination of the above memories.

[0220] The processor 901 is configured to execute any of the image annotation methods discussed above when calling the computer program stored in the memory 902. In addition, the processor 901 can also be configured to implement the above Figure 8 The functions of the device in the embodiment, or used to realize the functions of the terminal or server in the above text.

[0221] Based on the same inventive concept, an embodiment of the present application provides a computer storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer executes any one of the image annotation methods discussed above.

[0222] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0223] Based on the same inventive concept, embodiments of the present application provide a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image annotation method.

[0224] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other media that can store program codes.

[0225] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0226] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. An image annotation method, characterized in that: include: Determining similarities between the objects to be annotated in the image to be annotated and the annotated objects in each pre-stored annotated image, wherein each annotated image is annotated with a corresponding true category and each annotated image contains at least one annotated object; If the maximum value among the obtained similarities is less than a first threshold, obtaining an initial category of the image to be labeled according to at least one global feature of the image to be labeled; Clustering the images to be labeled into a dataset to be labeled corresponding to the initial categories; In response to a labeling operation performed on the dataset to be labeled, determining a true category to which each image to be labeled in the dataset to be labeled belongs, where the true category is a parent category to which the image to be labeled belongs; The step of determining the true category to which each image to be labeled in the dataset to be labeled belongs in response to the labeling operation performed on the dataset to be labeled includes: In response to a first selection operation triggered on the dataset to be annotated, displaying a plurality of groups included in the dataset to be annotated and a confirmation control associated with each group on an image annotation interface, wherein each group includes a plurality of images to be annotated; In response to a confirmation operation triggered by a confirmation control associated with a group among the multiple groups, determining that a true category of each to-be-annotated image in the group belongs to is the initial category; In response to a second selection operation triggered on the dataset to be annotated, displaying each image to be annotated and a corresponding plurality of sub-category labels in the image annotation interface, wherein the plurality of sub-category labels are a plurality of sub-category labels associated with the true category, and the plurality of sub-category labels indicate a next-level classification corresponding to the parent category; In response to each fourth selection operation triggered by a plurality of sub-category tags associated with an image to be annotated, the sub-category to which the image to be annotated belongs is determined to be the category corresponding to the selected sub-category tag.

2. The method according to claim 1, wherein The method further comprises: If the maximum value among the obtained similarities is not less than the first threshold, the category of the labeled image corresponding to the maximum value among the similarities is determined as the initial category of the image to be labeled.

3. The method according to claim 1, wherein Obtaining an initial category of the image to be labeled according to at least one global feature of the image to be labeled includes: Extracting at least one global feature of the image to be labeled; Inputting the at least one global feature into a trained classification model to obtain a probability that the image to be labeled belongs to each of a plurality of categories, wherein the trained classification model is obtained by training based on the labeled images; If the maximum value among the obtained multiple probabilities is not less than the second threshold, the category corresponding to the maximum value among the multiple probabilities is determined as the initial category of the image to be labeled.

4. The method according to claim 1, wherein Before determining the true category to which each to-be-annotated image in the to-be-annotated data set belongs in response to the labeling operation performed on the to-be-annotated data set, the method further includes: Displaying the dataset to be annotated and a plurality of candidate category labels associated with the dataset to be annotated in the image annotation interface, wherein the plurality of candidate category labels include a candidate category label corresponding to the initial category; In response to a first selection operation on any candidate category label among the multiple candidate category labels, the initial category of the to-be-labeled data set is updated to the category corresponding to the selected candidate category label.

5. The method according to claim 1, wherein When the first confirmation control associated with each group is displayed on the image annotation interface, the method further includes: displaying a first denial control associated with each group; The step of determining the true category to which each to-be-annotated image in the to-be-annotated dataset belongs in response to the labeling operation performed on the to-be-annotated dataset further includes: In response to a first deny operation triggered on a first deny control associated with a group among the plurality of groups, displaying a plurality of candidate category labels associated with the group; In response to a second selection operation triggered by multiple candidate category labels associated with the group, the true category of each to-be-annotated image in the group is determined to be the category corresponding to the selected candidate category label.

6. The method according to claim 1, wherein The method further comprises: When the image annotation interface displays a confirmation control associated with each group, a second denial control associated with each to-be-annotated image in each group is also displayed; In response to a second deny operation triggered by a second deny control associated with an image to be labeled, displaying a plurality of candidate category labels for the image to be labeled; In response to a third selection operation triggered by a plurality of candidate category labels associated with the image to be labeled, a true category of the image to be labeled is determined to be a category corresponding to the selected candidate category label.

7. An image annotation device, characterized in that: include: a determination module, configured to determine the similarity between the to-be-annotated object in the to-be-annotated image and the annotated objects in each pre-stored annotated image, wherein each annotated image is annotated with a corresponding true category and each annotated image contains at least one annotated object; an obtaining module, configured to obtain an initial category of the image to be labeled based on at least one global feature of the image to be labeled if the maximum value among the obtained similarities is less than a first threshold; A clustering module, configured to cluster the to-be-annotated images into to-be-annotated data sets corresponding to the initial categories; a labeling module, configured to determine, in response to a labeling operation performed on the dataset to be labeled, a true category to which each image to be labeled in the dataset to be labeled belongs, wherein the true category is a parent category to which the image to be labeled belongs; The step of determining the true category to which each image to be labeled in the dataset to be labeled belongs in response to the labeling operation performed on the dataset to be labeled includes: In response to a first selection operation triggered on the dataset to be annotated, displaying a plurality of groups included in the dataset to be annotated and a confirmation control associated with each group on an image annotation interface, wherein each group includes a plurality of images to be annotated; In response to a confirmation operation triggered by a confirmation control associated with a group among the multiple groups, determining that a true category of each to-be-annotated image in the group belongs to is the initial category; In response to a second selection operation triggered on the dataset to be annotated, displaying each image to be annotated and a corresponding plurality of sub-category labels in the image annotation interface, wherein the plurality of sub-category labels are a plurality of sub-category labels associated with the true category, and the plurality of sub-category labels indicate a next-level classification corresponding to the parent category; In response to each fourth selection operation triggered by a plurality of sub-category tags associated with an image to be annotated, the sub-category to which the image to be annotated belongs is determined to be the category corresponding to the selected sub-category tag.

8. The device according to claim 7, wherein The determining module is further configured to: If the maximum value among the obtained similarities is not less than the first threshold, the category of the labeled image corresponding to the maximum value among the similarities is determined as the initial category of the image to be labeled.

9. The device according to claim 7, wherein The acquisition module is specifically used for: Extracting at least one global feature of the image to be labeled; Inputting the at least one global feature into a trained classification model to obtain a probability that the image to be labeled belongs to each of a plurality of categories, wherein the trained classification model is obtained by training based on the labeled images; If the maximum value among the obtained multiple probabilities is not less than the second threshold, the category corresponding to the maximum value among the multiple probabilities is determined as the initial category of the image to be labeled.

10. A computer device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method according to any one of claims 1 to 6 by executing the instructions stored in the memory.

11. A computer storage medium, characterized in that The computer storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images