Model generation method, image annotation method, device and electronic device

Abstract

The embodiments of the present application disclose a model generation method, an image annotation method, a device, and an electronic device. The method includes: in the current training process, based on the first data set and the second data set, the image annotation model to be trained corresponding to the current training process is trained to obtain the trained model corresponding to the current training process; if the current training process meets the target training conditions, the trained model is used as the target image annotation model; if the current training process does not meet the target training conditions, the trained model is used as the image annotation model to be trained for the next training process. Through the above-mentioned method, the image annotation model to be trained is trained based on the first data set and the second data set to obtain the target image annotation model, thereby realizing the adaptation of the target image annotation model to the domain to which the second data set belongs, and improving the annotation accuracy of the target image annotation model for unlabeled images in the same domain as the second data set.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a model generation method, an image annotation method, a device, and an electronic device. Background Art

[0002] With the continuous development of artificial intelligence technology, image annotation has become a key task in the field of computer vision, with applications in areas such as image classification, keypoint detection, and person re-identification. Crowdsourcing and pre-trained model annotation are commonly used to annotate images. However, the accuracy of image annotation still needs to be improved. Summary of the Invention

[0003] In view of the above problems, the present application proposes a model generation method, an image annotation method, a device and an electronic device to improve the above problems.

[0004] In a first aspect, the present application provides a model generation method, which is applied to an electronic device, and the method includes: in a current training process, training an image annotation model to be trained corresponding to the current training process based on a first data set to obtain a first image annotation model corresponding to the current training process, the image annotation model to be trained includes a feature extraction network and a classifier network, and the first image annotation model has the same network structure as the image annotation model to be trained; training the classifier network of the first image annotation model based on the first data set and the second data set to obtain a second image annotation model corresponding to the current training process, the second image annotation model has the same network structure as the first image annotation model, and the first data set and the second data set belong to different fields; training the feature extraction network of the second image annotation model based on the second data set to obtain a trained model corresponding to the current training process, the trained model has the same network structure as the second image annotation model; if the current training process meets the target training conditions, the trained model is used as the target image annotation model; if the current training process does not meet the target training conditions, entering the next training process, and the trained model is used as the image annotation model to be trained corresponding to the next training process.

[0005] In a second aspect, the present application provides an image annotation method, which is applied to an electronic device. The method includes: obtaining an image to be annotated, where the image to be annotated belongs to the same field as the second data set; inputting the image to be annotated into a target image annotation model obtained based on the above method, and obtaining a label of the image to be annotated based on the predicted label output by the target image annotation model.

[0006] In a third aspect, the present application provides a model generation device that runs on an electronic device, the device comprising: a first image annotation model generation unit, for training the image annotation model to be trained corresponding to the current training process based on a first data set during the current training process, to obtain a first image annotation model corresponding to the current training process, the image annotation model to be trained comprising a feature extraction network and a classifier network, the first image annotation model having the same network structure as the image annotation model to be trained; a second image annotation model generation unit, for training the classifier network of the first image annotation model based on the first data set and the second data set, to obtain a second image annotation model corresponding to the current training process, the The second image annotation model has the same network structure as the first image annotation model, and the first data set and the second data set belong to different fields; the target image annotation model generation unit is used to train the feature extraction network of the second image annotation model based on the second data set to obtain a trained model corresponding to the current training process, and the trained model has the same network structure as the second image annotation model; if the current training process meets the target training conditions, the trained model is used as the target image annotation model; if the current training process does not meet the target training conditions, the next training process is entered, and the trained model is used as the image annotation model to be trained corresponding to the next training process.

[0007] In a fourth aspect, the present application provides an electronic device comprising one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0008] In a fifth aspect, the present application provides a computer-readable storage medium, in which program code is stored, wherein the above method is executed when the program code is run.

[0009] The present application provides a model generation method, image annotation method, device, electronic device and storage medium. During a training process, the image annotation model to be trained corresponding to the training process, including a feature extraction network and a classifier network, is trained based on a first data set to obtain a first image annotation model corresponding to the training process. The classifier network of the first image annotation model is then trained based on the first data set and a second data set to obtain a second image annotation model corresponding to the training process. The first data set and the second data set belong to different fields. The feature extraction network of the second image annotation model is then trained based on the second data set to obtain a trained model corresponding to the training process. If the training process meets the target training conditions, the trained model is used as the target image annotation model. If the training process does not meet the target training conditions, the next training process is entered, and the trained model is used as the image annotation model to be trained corresponding to the next training process. Through the above method, in each training process, the image annotation model to be trained can be trained step by step based on at least one of the first data set and the second data set, and the above training process can be repeated multiple times to obtain the target image annotation model, thereby realizing the adaptation of the image annotation model to be trained from the domain to which the first data set belongs to the domain described by the second data set, and improving the annotation accuracy of the target image annotation model for unlabeled images in the same domain as the second data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A flow chart of a model generation method proposed in an embodiment of the present application is shown;

[0012] Figure 2 A schematic diagram of an image annotation model to be trained proposed in this application is shown;

[0013] Figure 3 A schematic diagram of another image annotation model to be trained proposed in this application is shown;

[0014] Figure 4 Shows this application Figure 1 A flowchart of an embodiment of S120;

[0015] Figure 5 A schematic diagram of a first image annotation model proposed in this application is shown;

[0016] Figure 6 A schematic diagram of another first image annotation model proposed in this application is shown;

[0017] Figure 7 Shows this application Figure 1 A flowchart of an embodiment of S130;

[0018] Figure 8 A schematic diagram of the model training process proposed in this application is shown;

[0019] Figure 9 A flowchart of an image annotation method proposed in this application is shown;

[0020] Figure 10 A schematic diagram of a target image annotation model proposed in this application is shown;

[0021] Figure 11 A structural block diagram of a model generation device proposed in an embodiment of the present application is shown;

[0022] Figure 12 A structural block diagram of a structure search device proposed in an embodiment of the present application is shown;

[0023] Figure 13 Shows a structural block diagram of an electronic device proposed in this application;

[0024] Figure 14 It is a storage unit for storing or carrying program codes for implementing the model generation method and image annotation method according to the embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0026] With the continuous development of artificial intelligence technology, image annotation has become an important task in the field of computer vision, and can be applied to image classification, key point detection, pedestrian re-identification and other fields. Among the related methods, crowdsourcing annotation and pre-trained model annotation are commonly used to annotate images.

[0027] However, the inventors found in related research that the accuracy of image annotation needs to be improved. For example: the annotation quality of the crowdsourcing annotation method is heavily dependent on the quality of the annotation personnel, the annotation process and the annotation tools. In the crowdsourcing annotation process, the annotation personnel need to be familiar with the definitions of each annotation category and the annotation rules; at the same time, the annotation quality reviewers need to track the annotation status in a timely manner during the annotation process, otherwise there will be deviations in the understanding of the annotation personnel, resulting in a large number of unqualified annotation results, resulting in immeasurable annotation time and economic costs. For another example: although the annotation method based on the pre-trained model saves time and economic costs compared to the crowdsourcing annotation method, when the domain of the image to be annotated is significantly different from that of the training set of the pre-trained model, it may lead to inaccurate annotation.

[0028] Therefore, the inventors proposed a model generation method, image annotation method, device and electronic device in the present application. During the current training process, the image annotation model to be trained corresponding to the current training process, including a feature extraction network and a classifier network, is trained based on the first data set to obtain the first image annotation model corresponding to the current training process. The classifier network of the first image annotation model is then trained based on the first data set and the second data set to obtain the second image annotation model corresponding to the current training process. The first data set and the second data set belong to different fields. Then, the feature extraction network of the second image annotation model is trained based on the second data set to obtain the trained model corresponding to the current training process. If the current training process meets the target training conditions, the trained model is used as the target image annotation model. If the current training process does not meet the target training conditions, the next training process is entered, and the trained model is used as the image annotation model to be trained corresponding to the next training process. Through the above method, in each training process, the image annotation model to be trained can be trained in multiple steps based on at least one of the first data set and the second data set, and the above training process can be repeated multiple times to obtain the target image annotation model, thereby realizing the adaptation of the image annotation model to be trained from the domain to which the first data set belongs to the domain described by the second data set, and improving the annotation accuracy of the target image annotation model for unlabeled images in the same domain as the second data set.

[0029] See also Figure 1 The present application provides a model generation method, which is applied to electronic devices, and the method includes:

[0030] S110: During the current training process, the image annotation model to be trained corresponding to the current training process is trained based on the first data set to obtain a first image annotation model corresponding to the current training process, wherein the image annotation model to be trained includes a feature extraction network and a classifier network, and the network structure of the first image annotation model is the same as that of the image annotation model to be trained.

[0031] Wherein, the first data set may include multiple annotated images and real labels corresponding to each of the multiple annotated images, the real labels may represent the category information corresponding to each of the multiple annotated images, and the real labels may be obtained by manual annotation. Exemplarily, the first data set may be the open source data set ImageNet. The feature extraction network in the image annotation model to be trained may be used to map the first data set image to a deep feature space, and extract the deep features of the first data set image for image classification. Wherein, the structure of the feature extraction network may be various, such as: Alexnet, VGG16, ResNet and other network structures, which are not limited in the embodiments of the present application. The classifier network in the image annotation model to be trained may be used to classify images based on the features extracted by the feature extraction network, specifically, the image features representing the same category information may be aggregated as much as possible, and the distance between image features representing different category information may be expanded.

[0032] As a method, during the current training process, multiple annotated images can be input into the image annotation model to be trained corresponding to the current training process to obtain the predicted labels corresponding to each of the multiple annotated images; based on the difference between the predicted labels and the true labels, the loss value of the first loss function is determined, and the first loss function can be used to reduce the difference between the predicted labels and the true labels; based on the loss value, the model parameters of the image annotation model to be trained corresponding to the current training process are adjusted to obtain the first image annotation model corresponding to the current training process.

[0033] The first loss function may be a cross entropy loss function, a BCE loss function, etc., and the model parameters may be parameters that are variable during the model training process, such as weights, etc.

[0034] For example, Figure 2 As shown, during the current training process, the multiple annotated images of the first data set can be pre-processed and divided into multiple small batches (min-batches). Each batch can contain multiple annotated images (such as 32, 64, etc.) and input into the feature extraction network of the image annotation model to be trained corresponding to the current training process to extract image depth features. The extracted depth features are then input into the classifier network for classification processing and output corresponding prediction labels of the multiple annotated images. The prediction labels can represent the predicted classification results corresponding to the multiple annotated images. Therefore, the loss value of the first loss function can be determined based on the difference between the predicted labels and the true labels corresponding to the multiple annotated images. Then, the weight coefficient of the image annotation model to be trained corresponding to the current training process can be updated by backpropagation based on the loss value to obtain the first image annotation model corresponding to the current training process.

[0035] Optional, such as Figure 3 As shown, the classifier network may include a first classifier network and a second classifier network, and the predicted label includes a first predicted label and a second predicted label, wherein the first predicted label is the output result of the first classifier network, and the second predicted label is the output result of the second classifier network. As a method, the difference between the first predicted label and the true label corresponding to each of the multiple annotated images can be obtained as a first difference value; the difference between the second predicted label and the true label corresponding to each of the multiple annotated images can be obtained as a second difference value; based on the weighted sum of the first difference value and the second difference value, the loss value of the first loss function is determined. The calculation formula of the first loss function is as follows:

[0036] L C (X s , Y s )=L C1 (X s , Y s )+L C2 (X s , Y s )

[0037] Among them, X s The first dataset image, Y s It can represent the true label corresponding to the image of the first dataset, L C1 (X s , Y s ) can represent the difference between the first predicted label and the true label, L C2 (X s , Y s ) can represent the difference between the second predicted label and the true label. C1 (X s , Y s ), L C2 (X s , Y s ) can be obtained through the calculation formula of cross entropy loss function, BCE loss function, etc.

[0038] S120: Training the classifier network of the first image annotation model based on the first data set and the second data set to obtain a second image annotation model corresponding to the current training process. The second image annotation model has the same network structure as the first image annotation model, and the first data set and the second data set belong to different fields.

[0039] Among them, the domain of the image can be understood as the scene represented by the content of the image. Optionally, the domain of the image can be a combination of different attributes, such as: the content contained in the image (such as: human faces, animals, flowers, etc.), the brightness and darkness of the image (such as: natural images collected on sunny and cloudy days), the color of the image (such as: color images, grayscale images, etc.), the shooting angle of the image (such as: the front face and side face of the same person, etc.). In an embodiment of the present application, the domain to which the first data set belongs can be the source domain, and the domain to which the second data set belongs can be the target domain, wherein the image categories contained in the target domain can be the same as the image categories contained in the source domain, or the image categories contained in the target domain can be less than the image categories contained in the source domain.

[0040] Among them, the first classifier network and the second classifier network can respectively include a fully connected layer. Optionally, the fully connected layers of the first classifier network and the second classifier network can be the same, but the initialization parameters (such as: the initial weights of the fully connected layers, etc.) can be different. For example, the first classifier network and the second classifier network can both include three layers of fully connected layers, and the random initialization weight method is used to make the initialization parameters of the first classifier network and the second classifier network inconsistent. Optionally, the fully connected layers of the first classifier network and the second classifier network can also be different. For example, the number of layers of the fully connected layers of the first classifier network and the second classifier network is different.

[0041] As a way, Figure 4 As shown, the classifier network of the first image annotation model is trained based on the first data set and the second data set to obtain the second image annotation model corresponding to the training process, including:

[0042] S121: Input the multiple annotated images and the multiple unannotated images into the first image annotation model to obtain the predicted labels corresponding to each of the multiple annotated images, and the first predicted probability values and the second predicted probability values corresponding to each of the multiple unannotated images, wherein the first predicted probability value is output by the fully connected layer of the first classifier network, and the second predicted probability value is output by the fully connected layer of the second classifier network.

[0043] Among them, the first predicted probability value and the second predicted probability value can be a multidimensional vector, the vector dimension can be the same as the total number of categories in the first data set, and the value of each dimensional vector can represent the probability value of the unlabeled image being judged as a certain category by the classifier network.

[0044] As a way, Figure 5As shown, multiple annotated images of the first data set and multiple unannotated images of the second data set can be randomly mixed and input into the first image annotation model to obtain the predicted labels corresponding to each of the multiple annotated images, and the first predicted probability values and the second predicted probability values corresponding to each of the multiple unannotated images.

[0045] Optional, such as Figure 6 As shown, the outputs of the fully connected layers in the first classifier network and the second classifier network can be respectively used as inputs of a softmax activation function. Through the above method, the softmax function can normalize the outputs of the fully connected layers in the first classifier network and the second classifier network, and change the output range of the fully connected layer to between 0 and 1, so as to obtain the first predicted probability values and the second predicted probability values corresponding to each of the multiple unlabeled images and the multiple labeled images, and then the predicted labels corresponding to each of the multiple labeled images can be obtained based on the first predicted probability values and the second predicted probability values corresponding to each of the multiple labeled images.

[0046] S122: Based on the difference between the predicted labels and the true labels corresponding to each of the multiple labeled images, and the difference between the first predicted probability values and the second predicted probability values corresponding to each of the multiple unlabeled images, determine the loss value of a second loss function, wherein the second loss function is used to enable the first classification network and the second classifier network to correctly classify the first data set while increasing the difference between the first predicted probability values and the second predicted probability values corresponding to each of the multiple unlabeled images.

[0047] The second loss function may include a label loss and a probability loss. The label loss may be used to enable the first classifier network and the second classifier network to correctly classify the first data set. The calculation formula for the label loss may be the same as the calculation formula for the first loss function in step S110. The probability loss may be used to maximize the difference in the classification results of the first classifier network and the second classifier network for the second data set. The calculation formula for the probability loss may be:

[0048]

[0049] Among them, D t The second data set, x t can represent an image in the second dataset, θ F can represent the model parameters of the feature extraction network of the first image annotation model, may represent model parameters of the first classifier network of the first image annotation model, It can represent the model parameters of the second classifier network of the first image annotation model, F(x t θ F) can represent the deep features output by the feature extraction network of the first image annotation model, It can represent the first probability value output by the first classifier network of the first image annotation model, The second probability value may represent an output of the second classifier network of the first image annotation model.

[0050] Optionally, the probability loss may be the L1 distance, L2 distance, KL divergence, etc. between the first probability value and the second probability value. For example, when the probability loss is the L1 distance between the first probability value and the second probability value, The calculation formula can be:

[0051]

[0052] Where p1 can represent the first probability value output by the first classifier network of the first image annotation model; p2 can represent the second probability value output by the second classifier network of the first image annotation model; K can represent the dimensionality of the first probability value and the second probability value, that is, the number of labels in the second dataset; p 1k and p 2k They can respectively represent the probability values of the first classifier network and the second classifier network judging the second data set as the k-th label, that is, the value of the k-th dimension in the multidimensional vectors p1 and p2.

[0053] As a method, the label loss can be determined based on the difference between the predicted label and the true label corresponding to each of the multiple annotated images; the probability loss can be determined based on the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple annotated images; the probability loss is multiplied by a preset coefficient to obtain a reference probability loss, where the preset coefficient represents the importance of the probability loss; and the loss value of the second loss function can be determined based on the label loss and the reference probability loss. The calculation formula of the second loss function can be:

[0054] L2=L C (X s ,Y s )-α*L adv (X t )

[0055] Among them, the preset coefficient can be a fixed value between 0 and 1, which can be determined based on the results of multiple tests. If the probability loss contributes more to the training of the first image annotation model (reaching the local optimum of the second loss function as soon as possible), it can be indicated that the probability loss is more important, and the preset coefficient is larger at this time.

[0056] S123: Adjust the model parameters of the classifier network of the first image annotation model corresponding to the current training process based on the loss value to obtain the second image annotation model corresponding to the current training process, wherein, during the current training process, the model parameters of the feature extraction network of the first image annotation model remain unchanged.

[0057] Among them, as a method, when executing step S120, the model parameters of the feature extraction network of the first image annotation model can be frozen, and only the model parameters of the first classifier network and the second classifier network are updated based on the loss value of the second loss function to obtain the second image annotation model corresponding to the current training process.

[0058] S130: Training the feature extraction network of the second image annotation model based on the second data set to obtain a trained model corresponding to the current training process, wherein the trained model has the same network structure as the second image annotation model.

[0059] As a way, Figure 7 As shown, the feature extraction network of the second image annotation model is trained based on the second data set to obtain a trained model corresponding to the current training process, including:

[0060] S131: Input the multiple unlabeled images into the second image labeling model to obtain a first prediction probability value and a second prediction probability value corresponding to each of the multiple unlabeled images, wherein the first prediction probability value is output by the fully connected layer of the first classifier network, and the second prediction probability value is output by the fully connected layer of the second classifier network.

[0061] S132: Based on the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images, determine the loss value of a third loss function, wherein the third loss function is used to reduce the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images.

[0062] The third loss function may be the same as the calculation formula of the probability loss in step S122.

[0063] S133: Adjusting the model parameters of the feature extraction network of the second image annotation model based on the loss value to obtain a trained model corresponding to the current training process, wherein the model parameters of the classifier network of the second image annotation model remain unchanged during the current training process.

[0064] Among them, as a method, when executing step S130, the model parameters of the classifier network of the second image annotation model can be frozen, and only the model parameters of the feature extraction network can be updated based on the loss value of the third loss function to obtain the trained model corresponding to the current training process.

[0065] S140: If the current training process meets the target training conditions, the trained model is used as the target image annotation model; if the current training process does not meet the target training conditions, the next training process is entered, and the trained model is used as the image annotation model to be trained corresponding to the next training process.

[0066] Among them, the target training condition can be a preset number of training times. Under normal circumstances, when the preset number of training times continues to increase within a certain range, the accuracy of the target image annotation model obtained can be greatly improved, for example, the accuracy increases from 65% to 80%; when the preset number of training times exceeds this range, the accuracy of the target image annotation model obtained may be improved slightly, for example, from 80% to 80.5%.

[0067] Exemplarily, the target training condition can be set to a preset number of training times equal to 500 times. If the training process is the 300th repetition of steps S110 to S130, indicating that the target training condition has not been met, the trained model corresponding to the training process can be used as the 301st image annotation model to be trained; if the training process is the 500th repetition of steps S110 to S130, indicating that the target training condition has been met, the trained model corresponding to the training process can be used as the target image annotation model and the training is ended.

[0068] In the embodiment of the present application, by repeatedly executing steps S110 to S130, the first data set (source domain) and the second data set (target domain) can be mapped into a common feature space through feature extraction technology, and the image annotation model to be trained is trained multiple times through the second loss function and the third loss function to achieve feature alignment of the first data set (source domain) and the second data set (target domain) in the common feature space, thereby enabling the classifier network to correctly classify the first data set (source domain) and the second data set (target domain). For example, Figure 8 As shown, when the first data set and the second data set belong to different fields, by repeatedly executing steps S110 to S130, Figure 8 The image annotation model shown in the middle left is trained Figure 8 The target image annotation model shown in the middle right figure enables the model to move from a state where it can only correctly classify the first data set to a state where it can correctly classify both the first data set and the second data set, thereby improving Figure 8 The target image annotation model shown in the middle right figure has annotated accuracy for unlabeled images from the same domain as the second dataset.

[0069] This embodiment provides a model generation method. In the current training process, after training the image annotation model to be trained corresponding to the current training process, including a feature extraction network and a classifier network, based on a first data set, the first image annotation model corresponding to the current training process is obtained. Then, the classifier network of the first image annotation model is trained based on the first data set and the second data set to obtain the second image annotation model corresponding to the current training process. The first data set and the second data set belong to different fields. Then, based on the second data set, the feature extraction network of the second image annotation model is trained to obtain a trained model corresponding to the current training process. If the current training process meets the target training conditions, the trained model is used as the target image annotation model. If the current training process does not meet the target training conditions, the next training process is entered, and the trained model is used as the image annotation model to be trained corresponding to the next training process. Through the above method, in each training process, the image annotation model to be trained can be trained in multiple steps based on at least one of the first data set and the second data set, and the above training process can be repeated multiple times to obtain the target image annotation model, thereby realizing the adaptation of the image annotation model to be trained from the domain to which the first data set belongs to the domain described by the second data set, and improving the annotation accuracy of the target image annotation model for unlabeled images in the same domain as the second data set.

[0070] See also Figure 9 The present application provides an image annotation method, which is applied to electronic devices and includes:

[0071] S210: Acquire an image to be annotated, where the image to be annotated belongs to the same field as the second data set.

[0072] As one approach, images in the same field as that of the second dataset may be acquired based on an image acquisition device (such as a camera, a mobile phone, etc.) to obtain images to be annotated.

[0073] S220: Inputting the image to be annotated into the target image annotation model obtained based on the above method, and obtaining a label of the image to be annotated based on the predicted label output by the target image annotation model.

[0074] The target image annotation model may include a feature extraction network and a classifier network, the classifier network may include a first classifier network and a second classifier network, and the first classifier network and the second classifier network may respectively include a fully connected layer.

[0075] As a method, a first prediction value of the image to be labeled can be obtained based on the fully connected layer of the first classifier network; a second prediction value of the image to be labeled can be obtained based on the fully connected layer of the second classifier network; and a predicted label can be determined based on the first prediction value and the second prediction value to use the predicted label as the label of the image to be labeled.

[0076] Optional, such as Figure 10 As shown, the output of the fully connected layer in the first classifier network and the second classifier network is used as the input of an add layer, and the output of the add layer is used as the input of a softmax activation function, so that the first predicted value and the second predicted value can be added through the add layer, and the result of the addition is input into the softmax activation function, so that the softmax function can normalize the output of the fully connected layer in the first classifier network and the second classifier network, and change the output range of the fully connected layer to between 0 and 1, so as to obtain the predicted probability value of the target image annotation model for the image to be annotated, and take the label corresponding to the maximum value of the predicted probability value as the predicted label to obtain the label of the image to be annotated. The calculation formula of the predicted probability value is as follows:

[0077]

[0078] in, It can represent the model parameters of the feature extraction network of the target image annotation model, can represent the model parameters of the first classifier network of the target image annotation model, can represent the model parameters of the second classifier network of the target image annotation model, It can represent the deep features output by the feature extraction network of the target image annotation model, can represent the first probability value, A second probability value may be represented.

[0079] For example, the first probability value can be {0.25, 0.03, ..., 0.01}, the second probability value can be {0.35, 0.02, ..., 0.05}, and the predicted probability value can be {0.6, 0.05, ..., 0.06}, where the maximum value of the predicted probability value is 0.6, which indicates that the label corresponding to 0.6 is the label of the image to be annotated.

[0080] The image annotation method provided in this embodiment enables, through the above-mentioned method, to annotate images to be annotated in the same field as the second dataset based on a target image annotation model that is adaptive to the field of the second dataset, thereby improving the accuracy of image annotation.

[0081] See also Figure 11The present application provides a model generation device 600, which runs on an electronic device. The device 600 includes:

[0082] A first image annotation model generating unit 610 is configured to train, during the current training process, the image annotation model to be trained corresponding to the current training process based on the first data set to obtain a first image annotation model corresponding to the current training process, wherein the image annotation model to be trained includes a feature extraction network and a classifier network, and the first image annotation model has the same network structure as the image annotation model to be trained;

[0083] A second image annotation model generating unit 620 is configured to train the classifier network of the first image annotation model based on the first and second datasets to obtain a second image annotation model corresponding to the current training process, wherein the second image annotation model has the same network structure as the first image annotation model, and the first and second datasets belong to different fields;

[0084] The target image annotation model generation unit 630 is used to train the feature extraction network of the second image annotation model based on the second data set to obtain a trained model corresponding to the current training process, and the trained model has the same network structure as the second image annotation model; if the current training process meets the target training conditions, the trained model is used as the target image annotation model; if the current training process does not meet the target training conditions, the next training process is entered, and the trained model is used as the image annotation model to be trained corresponding to the next training process.

[0085] As a method, the first data set includes multiple annotated images and the real labels corresponding to each of the multiple annotated images. The first image annotation model generation unit 610 is specifically used to input the multiple annotated images into the image annotation model to be trained corresponding to the current training process during the current training process to obtain the predicted labels corresponding to each of the multiple annotated images; based on the difference between the predicted labels and the real labels, determine the loss value of the first loss function, and the first loss function is used to reduce the difference between the predicted labels and the real labels; based on the loss value, adjust the model parameters of the image annotation model to be trained corresponding to the current training process to obtain the first image annotation model corresponding to the current training process.

[0086] Optionally, the classifier network includes a first classifier network and a second classifier network, and the predicted label includes a first predicted label and a second predicted label, wherein the first predicted label is the output result of the first classifier network, and the second predicted label is the output result of the second classifier network. The first image annotation model generation unit 610 is specifically used to obtain the difference between the first predicted label and the true label as a first difference value; obtain the difference between the second predicted label and the true label as a second difference value; and determine the loss value of the first loss function based on the weighted sum of the first difference value and the second difference value.

[0087] As another embodiment, the second data set includes multiple unlabeled images, the classifier network includes a first classifier network and a second classifier network, the first classifier network and the second classifier network each include a fully connected layer, and the second image annotation model generation unit 620 Specifically, it is used to input the multiple labeled images and the multiple unlabeled images into the first image labeling model to obtain the predicted labels corresponding to each of the multiple labeled images, and the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images, wherein the first predicted probability value is output by the fully connected layer of the first classifier network, and the second predicted probability value is output by the fully connected layer of the second classifier network; based on the difference between the predicted labels corresponding to each of the multiple labeled images and the true labels, and the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images, determine the loss value of the second loss function, the second loss function is used to enable the first classification network and the second classifier network to correctly classify the first data set while increasing the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images; based on the loss value, adjust the model parameters of the classifier network of the first image labeling model corresponding to the current training process to obtain the second image labeling model corresponding to the current training process, wherein, during the current training process, the model parameters of the feature extraction network of the first image labeling model remain unchanged.

[0088] Optionally, the second loss function includes label loss and probability loss, and the second image annotation model generation unit 620 is specifically used to determine the label loss based on the difference between the predicted label corresponding to each of the multiple annotated images and the true label; determine the probability loss based on the difference between the first predicted probability value corresponding to each of the multiple annotated images and the second predicted probability value; multiply the probability loss by a preset coefficient to obtain a reference probability loss, and the preset coefficient represents the importance of the probability loss; and determine the loss value of the second loss function based on the label loss and the reference probability loss.

[0089] As another embodiment, the target image annotation model generation unit 630 is specifically used to input the multiple unlabeled images into the second image annotation model to obtain a first prediction probability value and a second prediction probability value corresponding to each of the multiple unlabeled images, wherein the first prediction probability value is output by the fully connected layer of the first classifier network, and the second prediction probability value is output by the fully connected layer of the second classifier network; based on the difference between the first prediction probability value and the second prediction probability value corresponding to each of the multiple unlabeled images, determine the loss value of the third loss function, and the third loss function is used to reduce the difference between the first prediction probability value and the second prediction probability value corresponding to each of the multiple unlabeled images; adjust the model parameters of the feature extraction network of the second image annotation model based on the loss value to obtain the trained model corresponding to the current training process, wherein, during the current training process, the model parameters of the classifier network of the second image annotation model remain unchanged.

[0090] See also Figure 12 The present application provides an image annotation device 800, which runs on an electronic device. The device 800 includes:

[0091] An image to be annotated acquiring unit 810 is configured to acquire an image to be annotated, where the image to be annotated belongs to the same field as the second dataset;

[0092] The labeling result acquisition unit 820 is configured to input the image to be labeled into the target image labeling model obtained based on the above method, and obtain the label of the image to be labeled based on the predicted label output by the target image labeling model.

[0093] As a method, the target image annotation model includes a feature extraction network and a classifier network, the classifier network includes a first classifier network and a second classifier network, the first classifier network and the second classifier network respectively include a fully connected layer, and the annotation result acquisition unit 820 is specifically used to obtain a first prediction value of the image to be annotated based on the fully connected layer of the first classifier network; obtain a second prediction value of the image to be annotated based on the fully connected layer of the second classifier network; determine the predicted label based on the first predicted value and the second predicted value, so as to use the predicted label as the label of the image to be annotated.

[0094] The following will be combined Figure 13 An electronic device provided by this application is described.

[0095] See also Figure 13Based on the above-mentioned model generation method, image annotation method, and apparatus, the embodiments of the present application further provide another electronic device 100 that can execute the above-mentioned model generation method and image annotation method. The electronic device 100 includes one or more (only one is shown in the figure) processors 102 and a memory 104 coupled to each other. The memory 104 stores a program that can execute the content of the above-mentioned embodiments, and the processor 102 can execute the program stored in the memory 104.

[0096] The processor 102 may include one or more processing cores. The processor 102 utilizes various interfaces and circuits to connect various components within the electronic device 100. It executes instructions, programs, code sets, or instruction sets stored in the memory 104, and accesses data stored in the memory 104 to perform various functions and process data within the electronic device 100. Optionally, the processor 102 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 102 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 102 and may be implemented separately via a communication chip.

[0097] The memory 104 may include a random access memory (RAM) or a read-only memory (ROM). The memory 104 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal 100 during use (such as a phone book, audio and video data, chat history data), etc.

[0098] Please refer to Figure 14, which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 1000 stores program code, which can be called by a processor to execute the method described in the above method embodiment.

[0099] The computer-readable storage medium 1000 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 1000 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1000 has storage space for program code 1010 for executing any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program code 1010 can be compressed, for example, in a suitable form.

[0100] In summary, the present application provides a model generation method, image annotation method, device and electronic device. During the current training process, the image annotation model to be trained corresponding to the current training process, including a feature extraction network and a classifier network, is trained based on the first data set to obtain the first image annotation model corresponding to the current training process. The classifier network of the first image annotation model is then trained based on the first data set and the second data set to obtain the second image annotation model corresponding to the current training process. The first data set and the second data set belong to different fields. Then, the feature extraction network of the second image annotation model is trained based on the second data set to obtain the trained model corresponding to the current training process. If the current training process meets the target training conditions, the trained model is used as the target image annotation model. If the current training process does not meet the target training conditions, the next training process is entered, and the trained model is used as the image annotation model to be trained corresponding to the next training process. Through the above method, in each training process, the image annotation model to be trained can be trained in multiple steps based on at least one of the first data set and the second data set, and the above training process can be repeated multiple times to obtain the target image annotation model, thereby realizing the adaptation of the image annotation model to be trained from the domain to which the first data set belongs to the domain described by the second data set, and improving the annotation accuracy of the target image annotation model for unlabeled images in the same domain as the second data set.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model generation method, characterized in that: Applied to electronic equipment, the method includes: During the current training process, the image annotation model to be trained corresponding to the current training process is trained based on the first data set to obtain the first image annotation model corresponding to the current training process, the image annotation model to be trained includes a feature extraction network and a classifier network, and the first image annotation model has the same network structure as the image annotation model to be trained; wherein the first data set includes multiple annotated images and the real labels corresponding to each of the multiple annotated images, and during the current training process, the image annotation model to be trained corresponding to the current training process is trained based on the first data set to obtain the first image annotation model corresponding to the current training process, including: during the current training process, the multiple annotated images are input into the image annotation model to be trained corresponding to the current training process to obtain the predicted labels corresponding to each of the multiple annotated images; based on the difference between the predicted labels and the real labels, the loss value of the first loss function is determined, the first loss function is used to reduce the difference between the predicted labels and the real labels; based on the loss value of the first loss function, the model parameters of the image annotation model to be trained corresponding to the current training process are adjusted to obtain the first image annotation model corresponding to the current training process; The classifier network of the first image annotation model is trained based on the first data set and the second data set to obtain the second image annotation model corresponding to the current training process, the second image annotation model has the same network structure as the first image annotation model, and the first data set and the second data set belong to different fields; the second data set includes multiple unlabeled images, the classifier network includes a first classifier network and a second classifier network, the first classifier network and the second classifier network respectively include a fully connected layer, and the classifier network of the first image annotation model is trained based on the first data set and the second data set to obtain the second image annotation model corresponding to the current training process, including: inputting the multiple labeled images and the multiple unlabeled images into the first image annotation model, obtaining the predicted labels corresponding to the multiple labeled images, the first predicted probability values and the second predicted probability values corresponding to the multiple unlabeled images. rate value, wherein the first predicted probability value is output by the fully connected layer of the first classifier network, and the second predicted probability value is output by the fully connected layer of the second classifier network; based on the difference between the predicted label and the true label corresponding to each of the multiple labeled images, and the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images, determine the loss value of the second loss function, the second loss function is used to enable the first classification network and the second classifier network to correctly classify the first data set while increasing the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images; based on the loss value of the second loss function, adjust the model parameters of the classifier network of the first image annotation model corresponding to the current training process to obtain the second image annotation model corresponding to the current training process, wherein, during the current training process, the model parameters of the feature extraction network of the first image annotation model remain unchanged; Inputting the plurality of unlabeled images into the second image labeling model to obtain a first predicted probability value and a second predicted probability value corresponding to each of the plurality of unlabeled images, wherein the first predicted probability value is output by the fully connected layer of the first classifier network, and the second predicted probability value is output by the fully connected layer of the second classifier network; determining a loss value of a third loss function based on the difference between the first predicted probability value and the second predicted probability value corresponding to each of the plurality of unlabeled images, wherein the third loss function is used to reduce the difference between the first predicted probability value and the second predicted probability value corresponding to each of the plurality of unlabeled images; adjusting the model parameters of the feature extraction network of the second image labeling model based on the loss value of the third loss function to obtain a trained model corresponding to the current training process, wherein during the current training process, the model parameters of the classifier network of the second image labeling model remain unchanged, and the trained model has the same network structure as the second image labeling model; If the current training process meets the target training conditions, the trained model will be used as the target image annotation model. If the current training process does not meet the target training conditions, the next training process will be entered, and the trained model will be used as the image annotation model to be trained corresponding to the next training process.

2. The method according to claim 1, characterized in that The classifier network includes a first classifier network and a second classifier network, the predicted label includes a first predicted label and a second predicted label, wherein the first predicted label is an output result of the first classifier network, and the second predicted label is an output result of the second classifier network, and determining the loss value of the first loss function based on the difference between the predicted label and the true label includes: Obtaining a difference between the first predicted label and the true label as a first difference value; Obtaining a difference between the second predicted label and the true label as a second difference value; A loss value of the first loss function is determined based on a weighted sum of the first difference value and the second difference value.

3. The method according to claim 1, characterized in that The second loss function includes a label loss and a probability loss, and determining the loss value of the second loss function based on the difference between the predicted label and the true label corresponding to each of the multiple labeled images, and the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images, includes: Determining the label loss based on a difference between the predicted labels corresponding to each of the plurality of annotated images and the true labels; determining the probability loss based on a difference between the first predicted probability value and the second predicted probability value corresponding to each of the plurality of annotated images; Multiplying the probability loss by a preset coefficient to obtain a reference probability loss, wherein the preset coefficient represents the importance of the probability loss; Based on the label loss and the reference probability loss, a loss value of the second loss function is determined.

4. An image annotation method, characterized in that: Applied to electronic equipment, the method includes: Acquire an image to be annotated, where the image to be annotated and the second data set belong to the same field; The image to be labeled is input into a target image labeling model obtained based on the method of any one of claims 1 to 3, and a label of the image to be labeled is obtained based on a predicted label output by the target image labeling model.

5. The method according to claim 4, characterized in that The target image annotation model includes a feature extraction network and a classifier network, the classifier network includes a first classifier network and a second classifier network, the first classifier network and the second classifier network each include a fully connected layer, and the label of the image to be annotated is obtained based on the predicted label output by the target image annotation model, including: a first prediction value of the image to be labeled obtained based on a fully connected layer of the first classifier network; a second prediction value of the image to be labeled obtained based on the fully connected layer of the second classifier network; The predicted label is determined based on the first predicted value and the second predicted value, so as to use the predicted label as a label of the image to be labeled.

6. A model generation device, characterized in that: Running on an electronic device, the device includes: A first image annotation model generating unit is configured to train the image annotation model to be trained corresponding to the current training process based on the first data set during the current training process to obtain the first image annotation model corresponding to the current training process, wherein the image annotation model to be trained includes a feature extraction network and a classifier network, and the network structure of the first image annotation model is the same as that of the image annotation model to be trained; wherein the first data set includes multiple annotated images and true labels corresponding to each of the multiple annotated images; and the image annotation model to be trained corresponding to the current training process based on the first data set during the current training process to obtain the first image annotation model corresponding to the current training process includes: during the current training process, inputting the multiple annotated images into the image annotation model to be trained corresponding to the current training process to obtain predicted labels corresponding to each of the multiple annotated images; determining a loss value of a first loss function based on the difference between the predicted labels and the true labels, wherein the first loss function is used to reduce the difference between the predicted labels and the true labels; and adjusting model parameters of the image annotation model to be trained corresponding to the current training process based on the loss value of the first loss function to obtain the first image annotation model corresponding to the current training process; The second image annotation model generating unit is used to train the classifier network of the first image annotation model based on the first data set and the second data set to obtain the second image annotation model corresponding to the current training process, the second image annotation model has the same network structure as the first image annotation model, and the first data set and the second data set belong to different fields; the second data set includes multiple unlabeled images, the classifier network includes a first classifier network and a second classifier network, the first classifier network and the second classifier network respectively include a fully connected layer, the classifier network of the first image annotation model is trained based on the first data set and the second data set to obtain the second image annotation model corresponding to the current training process, including: inputting the multiple labeled images and the multiple unlabeled images into the first image annotation model, obtaining the predicted labels corresponding to each of the multiple labeled images, the first predicted probabilities corresponding to each of the multiple unlabeled images a rate value and a second predicted probability value, wherein the first predicted probability value is output by the fully connected layer of the first classifier network, and the second predicted probability value is output by the fully connected layer of the second classifier network; based on the difference between the predicted label and the true label corresponding to each of the multiple labeled images, and the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images, determining the loss value of the second loss function, the second loss function is used to enable the first classification network and the second classifier network to correctly classify the first data set while increasing the difference between the first predicted probability value and the second predicted probability value corresponding to each of the multiple unlabeled images; adjusting the model parameters of the classifier network of the first image annotation model corresponding to the current training process based on the loss value of the second loss function to obtain the second image annotation model corresponding to the current training process, wherein, during the current training process, the model parameters of the feature extraction network of the first image annotation model remain unchanged; A target image annotation model generation unit is configured to input the plurality of unlabeled images into the second image annotation model to obtain a first predicted probability value and a second predicted probability value corresponding to each of the plurality of unlabeled images, wherein the first predicted probability value is output by the fully connected layer of the first classifier network, and the second predicted probability value is output by the fully connected layer of the second classifier network; based on the difference between the first predicted probability value and the second predicted probability value corresponding to each of the plurality of unlabeled images, determine a loss value of a third loss function, wherein the third loss function is configured to reduce the difference between the first predicted probability value and the second predicted probability value corresponding to each of the plurality of unlabeled images. difference; based on the loss value of the third loss function, the model parameters of the feature extraction network of the second image annotation model are adjusted to obtain the trained model corresponding to the current training process, wherein, in the current training process, the model parameters of the classifier network of the second image annotation model remain unchanged, and the trained model has the same network structure as the second image annotation model; if the current training process meets the target training conditions, the trained model is used as the target image annotation model; if the current training process does not meet the target training conditions, the next training process is entered, and the trained model is used as the image annotation model to be trained corresponding to the next training process.

7. An electronic device, characterized in that: including one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program code, wherein when the program code is run, the method according to any one of claims 1 to 4 is executed.

Images