Method and system for automatic image annotation

Abstract

The present invention discloses a method and system for automatic image annotation, characterized by comprising the following steps: acquiring images and placing them into a pool of images to be classified, screening and classifying the images; introducing new image categories and reference images based on the needs of the business model; calculating feature vectors of the reference images and the images to be classified; calculating the similarity between the images to be classified and the reference images, and automatically assigning the images to be classified to the category containing the reference image with the highest similarity; batch reviewing the results of the automatic machine classification to obtain a confirmed image dataset; and training a new model based on the confirmed image dataset. The present invention can efficiently and accurately complete image annotation, significantly reducing labor costs and accelerating the implementation of artificial intelligence applications.

Description

Technical Field

[0001] The present invention relates to the field of computer deep learning, and in particular to a method and system for automatic image annotation. Background Art

[0002] Artificial intelligence applications are increasing. However, deep learning requires a large amount of high-quality labeled data, which is a very time-consuming task. The most common method currently is to hire a large number of labelers or outsource the labeling to labeling companies, which is very costly.

[0003] Patent CN202010385771 (a method for annotating image data) assigns the annotation, review, and management of images to annotators, reviewers, and administrators respectively, and coordinates the work of all parties through annotation software. However, in essence, it still relies on manpower for annotation and can no longer meet the growing demand for annotation.

[0004] Patent CN201810400584.X (a crowdsourcing-based image annotation system) crowdsources the annotation work to volunteers. Although it can shorten the annotation cycle through multi-person collaboration, it does not reduce the annotation workload and is time-consuming and labor-intensive.

[0005] Patent CN202010355551 (Zero-sample image recognition method based on attribute feature vector and reversible generative model) uses an image dataset of known categories to train a reversible generative model, uses the model to generate new image data, and trains an SVM classifier based on this. However, it can be seen in step S1-2 that manual labeling is still required.

[0006] Therefore, an efficient method and system are needed to improve the efficiency of annotation and thus speed up the iteration of business models. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a method and system for automatic image annotation.

[0008] In actual business implementation, there is often a need to identify image categories, or to improve the accuracy of a certain category, or to increase the recognition capability of new categories. At this time, we need to add new high-quality data for training the image classification model.

[0009] To address the above technical issues, the present invention provides a method and system for automatic image annotation. Based on preset categories, reference images, and a deep learning model, the method calculates feature vectors of the image to be classified and the reference image. The image to be classified is automatically assigned to the category of the most similar reference image, achieving automatic image annotation. A simple batch review process then yields a high-quality image dataset. This method and system efficiently and accurately completes image annotation, significantly reducing labor costs and accelerating the implementation of artificial intelligence applications.

[0010] The present invention discloses a method for automatic image annotation, which is characterized by comprising the following steps:

[0011] Step 1: According to the needs of the business model, obtain images from the Internet and put them into the pool of images to be classified. The images to be classified are recorded as UIMG. The business model is a deep learning model Ci=F(IMG) that automatically classifies images according to business scenarios, where i represents the image category number, i=1,...,j;

[0012] Step 2: Introduce new image categories and reference images according to the needs of the business model. Specifically, define a new image category Ci', where i' represents the image category number, i'=1,...,t, and specify s reference images IMGk. The feature vector of the reference image is recorded as Vk, where k represents the reference image number, k=1,...,s;

[0013] Step 3: Calculate the feature vectors of the reference image and the image to be classified. Specifically, calculate the feature vector Vk of the reference image and the feature vector UV of the image to be classified.

[0014] Step 4: Calculate the similarity between the image to be classified and the reference image, and automatically assign the image to be classified to the category of the reference image with the highest similarity.

[0015] Step 5: Batch review the results of automatic machine classification to obtain a confirmed image dataset;

[0016] Step 6: Based on the confirmed image dataset, train the image classification model and evaluate it.

[0017] In the step 1, the requirements of the business model include: adding j+1, ..., j+t, a total of t image categories, and improving the recognition accuracy of category Ck.

[0018] In step 4, for the image to be classified, similarity is calculated between it and the reference image, and the categories of the top k reference images with the highest similarity are selected. The category with the largest number among the k categories is recorded as Cn, and its proportion is: p = number of Cn / k.

[0019] In step 5, based on the categories counted in step 4, a category proportion threshold th1 is set. If p>=th1, the image to be classified UIMG is assigned to the Cn category. This iteration is performed through automatic machine classification for rapid batch review to obtain a confirmed image dataset (C, IMG).

[0020] In step six, for each round of iterative image dataset, the corresponding image classification model Fk is trained using a deep learning network, and its accuracy is evaluated as TH(k). If TH(k)>Th(k+1) and TH(k)>Th(k+2), the iteration is stopped.

[0021] A system for automatic image annotation, characterized by comprising: a data source module, an automatic annotation module, and a model iteration module connected in sequence;

[0022] The data source module is used to prepare the data set required for training the artificial intelligence model, including a picture acquisition module for acquiring pictures;

[0023] The automatic annotation module is used for automatic image annotation; it includes five submodules connected in sequence: a new category introduction submodule, an image feature calculation submodule, an image similarity calculation submodule, an image automatic classification submodule, and an image batch review submodule;

[0024] The new category introduces a submodule for defining a new image classification and specifying reference images under the category;

[0025] The picture feature calculation submodule is used to calculate the feature vectors of the reference picture and the picture to be classified respectively;

[0026] The picture similarity calculation submodule is used to calculate the similarity between the picture to be classified and the reference picture;

[0027] The automatic image classification submodule is used to automatically classify the image to be classified into the category of the most similar reference image;

[0028] The image batch review submodule is used to batch review the results of automatic machine classification to obtain a confirmed image data set;

[0029] The model iteration module is used for iteration of the artificial intelligence model, including a connected model training submodule and a model evaluation submodule;

[0030] The model training submodule is used to train a new model based on the newly confirmed image dataset;

[0031] The model evaluation submodule is used to evaluate the accuracy of the newly trained model. If the accuracy does not meet the requirements, the above steps will continue to be repeated.

[0032] The beneficial effects achieved by the present invention include the design of a method and system for automatic image annotation to address the cost of annotation. Based on preset categories, reference images, and a deep learning model, the method calculates the feature vectors of the image to be classified and the reference image, automatically assigning the image to the category of the most similar reference image. This enables automatic image annotation, and a high-quality image dataset can be obtained through simple batch review. This method and system can efficiently and accurately complete image annotation, significantly reducing labor costs and accelerating the implementation of artificial intelligence applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A simplified flowchart of a method for automatically annotating pictures according to an exemplary embodiment of the present invention;

[0034] Figure 2 Schematic diagram of the structure of an automatic image annotation system in an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention discloses a method for automatic image annotation, comprising:

[0036] Step 1: Get pictures from the Internet and put them into the pool of pictures to be classified. In actual business implementation, there is often a need to identify picture categories, or to improve the accuracy of a certain category, or to increase the recognition ability of new categories. At this time, we need to add new high-quality data for training the picture classification model. According to the needs of the business model, get pictures from the Internet and put them into the pool of pictures to be classified, and screen and classify the pictures; the business model refers to a deep learning model Ci=F(IMG) that automatically classifies pictures according to business scenarios, i represents the number of the picture category, i=1,...,j, and the needs of the business model include: adding j+1,...,j+t, a total of t picture categories or improving the recognition accuracy of category Ck. At this time, we need to collect a batch of new high-quality data for model upgrades.

[0037] Step 2: Introduce new categories and reference images based on demand. Based on the business model requirements, define a new image category Ci', where i' represents the image category number, i' = 1, ..., t, and specify s reference images IMGk, where k = 1, ..., s.

[0038] Step 3: Calculate the feature vectors of the reference image and the image to be classified. Calculate the feature vector Vk of the reference image, k = 1, ..., s; for the image to be classified UIMG, calculate its image feature vector UV.

[0039] Step 4: Calculate the similarity between the image to be classified and the reference image, and automatically assign the image to be classified to the category of the reference image with the highest similarity. Specifically, for the image to be classified, calculate the similarity between it and the classified images, and take the top k categories of the classified images with the highest similarity. Count the category with the largest number among the k categories as Cn, and its proportion is p = number of Cn / k.

[0040] Step 5: Batch review the results of the machine automatic classification to obtain a confirmed image dataset. Specifically, set the category proportion threshold th1. If p>=th1, the image to be classified UIMG is assigned to category Cn. This iterative process uses machine automatic classification to quickly conduct batch review and quickly obtain a confirmed image dataset (C, IMG).

[0041] Step 6: Based on the confirmed image dataset, train the image classification model and evaluate it. Specifically, for each round of iterative image dataset, use the deep learning network to train the corresponding image classification model Fk=F(IMG), and evaluate its accuracy as TH(k), TH(k)>Th(k+1) and TH(k)>Th(k+2), then stop the iteration.

[0042] The invention discloses a system for automatic image annotation, comprising: a data source module, an automatic annotation module, and a model iteration module.

[0043] The data source module is used to prepare the data set required for training the artificial intelligence model, including an image acquisition module.

[0044] The automatic labeling module is used for automatic image labeling. It includes five submodules: a new category introduction submodule, an image feature calculation submodule, an image similarity calculation submodule, an image automatic classification submodule, and an image batch review submodule. The new category introduction submodule is used to define a new image category and specify the reference image under the category. The image feature calculation submodule calculates the feature vectors of the reference image and the image to be classified respectively. The image similarity calculation submodule calculates the similarity between the image to be classified and the reference image. The image automatic classification submodule automatically classifies the image to be classified into the category of the most similar reference image. The image batch review submodule batch reviews the results of automatic machine classification to obtain a confirmed image data set.

[0045] The model iteration module, used for iteration of the AI ​​model, includes a model training submodule and a model evaluation submodule. The model training submodule trains a new model based on the newly confirmed image dataset. The model evaluation submodule evaluates the accuracy of the newly trained model and repeats the above steps if it does not meet the requirements.

[0046] The present invention will be further described below with reference to the accompanying drawings and exemplary embodiments:

[0047] See Figure 1 As shown, the process of the embodiment of the present invention includes the following specific steps:

[0048] Step 11: Get images from the internet and put them into the pool of images to be classified. Based on the business model requirements, relevant images are captured from the internet. However, since only a portion of the captured images meet the actual requirements, image screening and classification are required.

[0049] Step 12: Introduce new categories and reference images as needed. Based on the business model requirements, define t new image categories Ci', where i' represents the image category number, i' = 1, ..., t, and specify s reference images IMGk, where k = 1, ..., s.

[0050] Step 13: Calculate the feature vectors of the reference image and the image to be classified. Calculate the feature vector Vk of the reference image, k = 1, ..., s; and calculate the image feature vector UV of the image to be classified UIMG.

[0051] Step 14: Calculate the similarity between the image to be classified and the reference image, and automatically assign the image to be classified to the category of the reference image with the highest similarity. Specifically, for the image to be classified, calculate its similarity with the classified images, and take the top k categories of the classified images with the highest similarity. Count the category with the largest number among the k categories as Cn, and its proportion is p = number of Cn / k.

[0052] Step 15: Batch review the results of the machine automatic classification to obtain a confirmed image dataset. Specifically, set the category proportion threshold th1. If p>=th1, the image to be classified UIMG is assigned to category Cn. This iterative process uses machine automatic classification to quickly conduct batch review and quickly obtain a confirmed image dataset (C, IMG).

[0053] Step 16: Based on the confirmed image dataset, train the image classification model and evaluate it. Specifically, for each round of iterative image dataset, use the deep learning network to train the corresponding image classification model Fk, and evaluate its accuracy as TH(k). If TH(k)>Th(k+1) and TH(k)>Th(k+2), the iteration can be stopped.

[0054] Deep neural networks are a technology in the field of machine learning (ML). The advantage of multiple layers is that complex functions can be represented with fewer parameters. In supervised learning, the problem with previous multi-layer neural networks is that they are prone to falling into local extreme points. If the training samples sufficiently cover future samples, then the learned multi-layer weights can be used to predict new test samples. Hinton proposed an effective method for building multi-layer neural networks on unsupervised data. Simply put, it is divided into two steps. The first is to train the network one layer at a time, and the second is to tune so that the high-level representation r generated by the original representation x and the x' generated by the high-level representation r are as consistent as possible. The method is:

[0055] 1. First, build a single layer of neurons layer by layer, so that a single layer network is trained each time.

[0056] 2. After all layers are trained, Hinton uses the wake-sleep algorithm for tuning. The weights between layers other than the top layer are made bidirectional, so that the top layer is still a single-layer neural network, while the other layers become graphical models. The upward weights are used for "cognition" and the downward weights are used for "generation". Then the Wake-Sleep algorithm is used to adjust all weights. To make cognition and generation consistent, that is, to ensure that the generated top-level representation can restore the bottom-level nodes as accurately as possible. For example, if a node at the top layer represents a face, then all images of a face should activate this node, and the image generated downward from this result should be able to appear as an approximate face image. The Wake-Sleep algorithm is divided into two parts: wake and sleep.

[0057] 2.1, the awakening phase, the cognitive process, generates abstract representations (node ​​states) at each layer using external features and upward weights (cognitive weights). Gradient descent is then used to modify downward weights between layers (generative weights). In other words, "If reality is different from what I imagined, change my weights to make it what I imagined."

[0058] 2.2, the sleep phase, is the generative process. This involves generating the underlying states through top-level representations (concepts learned while awake) and downward weights, while also modifying the upward weights between layers. In other words, if the dream image isn't the corresponding concept in my mind, change my cognitive weights so that it appears to be the concept.

[0059] Since the autoencoder (the neural network mentioned above. In a broad sense, the autoencoder refers to all structures that obtain high-level representations from low-level representations and can generate approximate low-level representations from high-level representations, and in a narrow sense, it refers to one of them, used by Google's face recognition) has an associative function, that is, missing parts of the input can also be correctly encoded, so the algorithm mentioned above can also be used for supervised learning. During training, y is used as a supplement to the top-level network input, and during application, the top-level network generates y'.

[0060] See Figure 2 As shown, the structure of a system embodiment of the present invention includes: a data source module 21, an automatic labeling module 22, and a model iteration module 23.

[0061] The data source module 21 is used to prepare the data set required for training the artificial intelligence model, including the image acquisition module 211.

[0062] The automatic annotation module 22 is used for automatic image annotation and includes five submodules: a new category introduction submodule 221 , an image feature calculation submodule 222 , an image similarity calculation submodule 223 , an image automatic classification submodule 224 , and an image batch review submodule 225 .

[0063] The new category introduction submodule 221 is used to define a new picture category and specify reference pictures under the category.

[0064] The picture feature calculation submodule 222 calculates feature vectors of the reference picture and the picture to be classified respectively.

[0065] The picture similarity calculation submodule 223 calculates the similarity between the picture to be classified and the reference picture.

[0066] The automatic picture classification submodule 224 automatically classifies the picture to be classified into the category of the most similar reference picture.

[0067] The picture batch review submodule 225 reviews the results of the automatic classification by the machine in batches to obtain a confirmed picture data set.

[0068] The model iteration module 23 is used for iteration of the artificial intelligence model, and includes a model training submodule 231 and a model evaluation submodule 232.

[0069] The model training submodule 231 trains a new model based on the newly confirmed image dataset.

[0070] The model evaluation submodule 232 performs an accuracy evaluation on the newly trained model, and if the accuracy does not meet the requirements, the above steps are continued.

[0071] This invention primarily provides a method and system for automatic image annotation. Based on preset categories, reference images, and a deep learning model, the method calculates feature vectors of the image to be classified and the reference image. The image to be classified is automatically assigned to the category of the most similar reference image, enabling automatic image annotation. A simple batch review process yields a high-quality image dataset. This method and system can efficiently and accurately complete image annotation, significantly reducing labor costs and accelerating the implementation of artificial intelligence applications.

[0072] The above embodiments do not limit the present invention in any way. Any other improvements and applications made to the above embodiments in an equivalent manner fall within the scope of protection of the present invention.

Claims

1. A method for automatic image annotation, characterized in that: The steps include: Step 1: According to the requirements of the business model, obtain images from the Internet and put them into the pool of images to be classified. The images are screened and classified. The images to be classified are recorded as UIMG. The business model is a deep learning model Ci=F(IMG) that automatically classifies images according to business scenarios, where i represents the number of the image category, i=1,...,j; Step 2: Introduce new image categories and reference images according to the needs of the business model. Specifically, define a new image category Ci', where i' represents the image category number, i'=1,...,t, and specify s reference images IMGk. The feature vector of the reference image is recorded as Vk, where k represents the reference image number, k=1,...,s; Step 3: Calculate the feature vectors of the reference image and the image to be classified. Specifically, calculate the feature vector Vk of the reference image and the feature vector UV of the image to be classified. Step 4: Calculate the similarity between the image to be classified and the reference image, and automatically assign the image to be classified to the category of the reference image with the highest similarity. For the image to be classified, calculate its similarity with the reference image, take the categories of the top k reference images with the highest similarity, and count the category with the largest number among the k categories as Cn, and its proportion is: p = number of Cn / k; Step 5: Batch review the results of automatic machine classification to obtain a confirmed image dataset; This includes: based on the categories counted in step 4, setting a category ratio threshold th1. If p>=th1, the image to be classified UIMG is assigned to category Cn. This process is repeated repeatedly to perform rapid batch review through machine automatic classification. Step 6: Based on the confirmed image dataset, train the image classification model and evaluate it; Including: for each round of iterative image dataset, use the deep learning network to train the corresponding image classification model Fk, and evaluate its accuracy as TH(k). If TH(k)>Th(k+1) and TH(k)>Th(k+2), the iteration is stopped.

2. The method for automatic image annotation according to claim 1, wherein: In the step 1, the requirements of the business model include: adding j+1, ..., j+t, a total of t image categories, and improving the recognition accuracy of category Ck.

3. A system for automatically annotating images according to the method of any one of claims 1-2, characterized in that: It includes the following modules: data source module, automatic annotation module, and model iteration module. The data source module is used to prepare the data set required for training the artificial intelligence model, including a picture acquisition module for acquiring pictures; The automatic annotation module is used for automatic image annotation; it includes five submodules connected in sequence: a new category introduction submodule, an image feature calculation submodule, an image similarity calculation submodule, an image automatic classification submodule, and an image batch review submodule; The new category introduces a submodule for defining a new image classification and specifying reference images under the category; The picture feature calculation submodule is used to calculate the feature vectors of the reference picture and the picture to be classified respectively; The picture similarity calculation submodule is used to calculate the similarity between the picture to be classified and the reference picture; The automatic image classification submodule is used to automatically classify the image to be classified into the category of the most similar reference image; The image batch review submodule is used to batch review the results of automatic machine classification to obtain a confirmed image data set; The model iteration module is used for iteration of the artificial intelligence model, including a connected model training submodule and a model evaluation submodule; The model training submodule is used to train a new model based on the newly confirmed image dataset; The model evaluation submodule is used to evaluate the accuracy of the newly trained model. If the accuracy does not meet the requirements, the above steps will continue to be repeated.

Images