Image label acquisition method, image search method, device and electronic equipment

Abstract

Embodiments of the present application disclose an image label acquisition method, an image search method, an image label acquisition device, an image search device, an electronic device and a storage medium. The method comprises: acquiring a global feature of a to-be-classified image; inputting the global feature into a classification prediction module of a multi-label classification model, acquiring a plurality of classification labels corresponding to the to-be-classified image output by the classification prediction module, and a feature map corresponding to each of the plurality of classification labels; inputting the feature map corresponding to each of the plurality of classification labels into an attribute prediction module of the multi-label classification model, and acquiring an attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module. Through the above method, the global feature of the to-be-classified image is directly recognized by the multi-label classification model, and then the plurality of classification labels corresponding to the to-be-classified image can be directly determined, the process of acquiring the image label is simplified, and the speed of acquiring the image label is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and particularly relates to an image label acquisition method, an image search method, an image label acquisition device, an electronic device and a storage medium. BACKGROUND

[0002] Image label acquisition is a common task in the field of image processing, and can help a model understand an image in a more fine-grained manner in a cross-modal field. Related image label acquisition methods are still relatively complex in the process of acquiring image labels. SUMMARY

[0003] In view of the above problems, the present application provides an image label acquisition method, an image search method, an image label acquisition device, an electronic device and a storage medium to improve the above problems.

[0004] In a first aspect, an image label acquisition method is provided, and the method comprises: acquiring a global feature of a to-be-classified image; inputting the global feature into a classification prediction module of a multi-label classification model, acquiring a plurality of classification labels corresponding to the to-be-classified image output by the classification prediction module, and a feature map corresponding to each of the plurality of classification labels; inputting the feature map corresponding to each of the plurality of classification labels into an attribute prediction module of the multi-label classification model, and acquiring an attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0005] In a second aspect, an image search method is provided, and the method comprises: acquiring an input preset label, the preset label comprising at least one of a classification label and an attribute label; comparing the preset label with a plurality of classification labels and attribute labels corresponding to each of images in an album; acquiring a target image corresponding to the preset label from the album, and displaying the target image.

[0006] In a third aspect, an image label acquisition device is provided, and the device comprises:

[0007] a feature acquisition unit configured to acquire a global feature of a to-be-classified image; a feature map acquisition unit configured to input the global feature into a classification prediction module of a multi-label classification model, acquire a plurality of classification labels corresponding to the to-be-classified image output by the classification prediction module, and a feature map corresponding to each of the plurality of classification labels; and an attribute acquisition unit configured to input the feature map corresponding to each of the plurality of classification labels into an attribute prediction module of the multi-label classification model, and acquire an attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0008] In a fourth aspect, an embodiment of the present application provides an image search device, the device comprising: a label obtaining unit configured to obtain an inputted preset label, the preset label comprising at least one of a classification label and an attribute label; a comparison unit configured to compare the preset label with a plurality of classification labels and attribute labels corresponding to each of images in an album; and a display unit configured to obtain a target image corresponding to the preset label from the album and display the target image.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the method described above.

[0010] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores program codes, and the program codes perform the method described above when the program codes run.

[0011] Embodiments of the present application provide an image label obtaining method, an image search method, a device, an electronic device and a storage medium. Firstly, the global feature of the image to be classified is obtained, then the global feature is input into the classification prediction module of the multi-label classification model, the plurality of classification labels corresponding to the image to be classified and the feature maps corresponding to the plurality of classification labels are obtained, and finally the feature maps corresponding to the plurality of classification labels are input into the attribute prediction module of the multi-label classification model, and the attribute labels corresponding to the plurality of classification labels are obtained. Through the above method, the global feature of the image to be classified is directly recognized through the multi-label classification model, and then the plurality of classification labels corresponding to the image to be classified and the feature maps corresponding to the plurality of classification labels can be directly determined. After the feature maps corresponding to the plurality of classification labels are determined, the attribute labels corresponding to each classification label can be quickly determined, the process of obtaining the image label is simplified, and the speed of obtaining the image label is improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0013] Figure 1 An application scenario schematic diagram of an image label obtaining method according to an embodiment of the present application is shown;

[0014] Figure 2 An application scenario of the image label acquisition method is shown;

[0015] Figure 3 A flow chart of the image label acquisition method is shown;

[0016] Figure 4 A flow chart of the image label acquisition method is shown;

[0017] Figure 5 An application scenario of the image label acquisition method is shown;

[0018] Figure 6 A flow chart of the image label acquisition method is shown;

[0019] Figure 7 A flow chart of the image label acquisition method is shown;

[0020] Figure 8 A flow chart of the image label acquisition method is shown;

[0021] Figure 9 An application scenario of the image label acquisition method is shown;

[0022] Figure 10 An application scenario of the image label acquisition method is shown;

[0023] Figure 11 An application scenario of the image label acquisition method is shown;

[0024] Figure 12 An application scenario of the image label acquisition method is shown;

[0025] Figure 13 An application scenario of the image label acquisition method is shown;

[0026] Figure 14 A structure block diagram of the image label acquisition device is shown;

[0027] Figure 15 A structure block diagram of the image label acquisition device is shown;

[0028] Figure 16 A structure block diagram of the image search device is shown;

[0029] Figure 17 A structural block diagram of an electronic device or a server for performing an image label obtaining method according to an embodiment of the present application is shown in the present application;

[0030] Figure 18 A storage unit for storing or carrying program code for implementing an image label obtaining method according to an embodiment of the present application is shown in the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0032] Image label obtaining is a common task in the field of image processing, which can help the model understand the image in a more fine-grained manner in the cross-modal field. Among them, the modal in the cross-modal refers to the existing form of data, such as text, audio, image, video, etc. Cross-modal refers to describing the same thing or event through different existing forms of data. For example, a cat can be described in the form of text and image.

[0033] The inventor found in the research on related image label obtaining methods that the related image label obtaining methods can first input an image into a convolutional neural network, obtain a feature map output by the convolutional neural network, then predict a candidate box using the feature map, combine the candidate box and the feature map to generate a region feature, and then obtain a label for the region feature. That is, in the process of obtaining an image label, the objects in the image need to be calibrated through a large number of candidate boxes first, then the region feature of the region where the candidate box is located is extracted, and finally the image label is obtained through the region feature. The above method classifies after detecting the objects in the image, which will lead to high redundancy of target candidate boxes, high model calculation complexity, and low efficiency.

[0034] Therefore, the inventors propose the image label obtaining method, the image search method, the device, the electronic equipment and the storage medium in the present application. Firstly, the global feature of the image to be classified is obtained, and then the global feature is input into a classification prediction module of a multi-label classification model to obtain a plurality of classification labels corresponding to the image to be classified and a feature map corresponding to each of the plurality of classification labels output by the classification prediction module. Finally, the feature map corresponding to each of the plurality of classification labels is input into an attribute prediction module of the multi-label classification model to obtain an attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module. Through the above method, the global feature of the image to be classified is directly recognized by the multi-label classification model, and then the plurality of classification labels corresponding to the image to be classified and the feature map corresponding to each of the plurality of classification labels can be directly determined. After the feature map corresponding to each of the plurality of classification labels is determined, the attribute label corresponding to each classification label can be quickly determined, the process of obtaining the image label is simplified, and the speed of obtaining the image label is improved.

[0035] In the embodiments of the present application, the image label obtaining method provided can be executed by an electronic equipment. In this way executed by the electronic equipment, all steps in the image label obtaining method provided by the embodiments of the present application can be executed by the electronic equipment. For example, as shown in Figure 1 the processor in the electronic equipment 100 executes to obtain the global feature of the image to be classified; inputs the global feature into a classification prediction module of a multi-label classification model to obtain a plurality of classification labels corresponding to the image to be classified and a feature map corresponding to each of the plurality of classification labels output by the classification prediction module; and inputs the feature map corresponding to each of the plurality of classification labels into an attribute prediction module of the multi-label classification model to obtain an attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0036] Furthermore, the image label obtaining method provided by the embodiments of the present application can also be executed by a server (cloud). Correspondingly, in this way executed by the server, the image to be classified can be obtained by the electronic equipment and synchronously sent to the server, and then the global feature of the image to be classified is obtained by the server in real time; the global feature is input into a classification prediction module of a multi-label classification model to obtain a plurality of classification labels corresponding to the image to be classified and a feature map corresponding to each of the plurality of classification labels output by the classification prediction module; and the feature map corresponding to each of the plurality of classification labels is input into an attribute prediction module of the multi-label classification model to obtain an attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0037] In addition, the image label obtaining method can be cooperatively executed by the electronic device and the server. In the cooperative execution by the electronic device and the server, part of the steps in the image label obtaining method is executed by the electronic device, and the other part of the steps is executed by the server.

[0038] For example, as shown in Figure 2 , the electronic device 100 can execute the image label obtaining method including: obtaining a global feature of a to-be-classified image, then executing, by the server 200, inputting the global feature into a classification prediction module of a multi-label classification model, obtaining a plurality of classification labels corresponding to the to-be-classified image and a feature map corresponding to each of the plurality of classification labels output by the classification prediction module; inputting the feature map corresponding to each of the plurality of classification labels into an attribute prediction module of the multi-label classification model, and obtaining an attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0039] It should be noted that, in the cooperative execution by the electronic device and the server, the steps executed by the electronic device and the server are not limited to the above-mentioned examples, and in actual application, the steps executed by the electronic device and the server can be dynamically adjusted according to actual conditions.

[0040] It should be noted that, in addition to the smart phone shown in Figure 1 and Figure 2 , the electronic device 100 can also be a car device, a wearable device, a tablet computer, a notebook computer, a smart speaker, etc. The server 120 can be a stand-alone physical server, or a server cluster or a distributed system composed of multiple physical servers.

[0041] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0042] Please refer to Figure 3 , the image label obtaining method provided by the embodiments of the present application is applied to an electronic device or a server as shown in Figure 1 or Figure 2 , and the method includes:

[0043] Step S110: obtaining a global feature of a to-be-classified image.

[0044] In the embodiments of the present application, the to-be-classified image can be understood as an image for which a corresponding label needs to be obtained. The to-be-classified image can be a real-time obtained image or a pre-obtained image. Optionally, the to-be-classified image can be an RGB image, a grayscale image or a YUV image, etc., which is not limited here.

[0045] As a manner, the image to be classified is an image acquired in real time. In this case, the image to be classified can be an image downloaded from a browser or a server in real time, or can be an image acquired in real time by an image acquisition device. The image acquisition device can be a smart device provided with a camera, such as a mobile phone, a camera, a notebook computer, etc.

[0046] As another manner, the image to be classified is an image acquired in advance. In this case, the image to be classified can be an image acquired in advance in response to a preset instruction. The preset instruction can be triggered when a preset application program is detected. The preset application program can be an application program in an electronic device for acquiring image tags.

[0047] Optionally, in the embodiment of the present application, the global feature of the image to be classified can be understood as the overall attribute of the image to be classified. The global feature of the image to be classified can include color features, texture features, shape features, etc. of the image to be classified.

[0048] As a manner, the global feature of the image to be classified can be extracted by a convolutional neural network. Specifically, the step of extracting the global feature of the image to be classified by the convolutional neural network can include: inputting the image to be classified into the convolutional neural network, and acquiring the global feature of the image to be classified output by the convolutional neural network. The convolutional neural network outputs a plurality of feature maps corresponding to the image to be classified, and the plurality of feature maps are taken as the global feature of the image to be classified.

[0049] Step S120: inputting the global feature into a classification prediction module of the multi-label classification model, acquiring a plurality of classification labels corresponding to the image to be classified and a feature map corresponding to each of the plurality of classification labels output by the classification prediction module.

[0050] In the embodiment of the present application, the multi-label classification model is a pre-trained model for predicting classification labels and attribute labels of the image to be classified. The classification label is a label corresponding to a target object included in the image to be classified, and is used to represent the category of the target object. All objects included in the image to be classified are target objects.

[0051] Optionally, in the embodiment of the present application, each classification label can correspond to a feature map. The plurality of feature maps output by the convolutional neural network are input into the classification prediction module of the multi-label classification model for classification label prediction, and the plurality of classification labels corresponding to the image to be classified and the feature map corresponding to each classification label output by the classification prediction module are acquired.

[0052] Step S130: inputting the feature map corresponding to each of the plurality of classification labels into an attribute prediction module of the multi-label classification model, to obtain the attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0053] In the embodiment of the present application, the attribute label is used to represent the self attribute of the target object, such as color, quantity, etc. The feature map corresponding to each classification label is input into the attribute prediction module of the multi-label classification model for attribute label prediction, and the attribute label corresponding to each classification label output by the attribute prediction module is obtained. The output of the classification prediction module of the multi-label classification model is used as the input of the attribute prediction module of the multi-label classification model. When the attribute prediction module outputs the attribute label corresponding to each classification label, not every classification label will have an attribute label. If the feature map corresponding to a certain classification label is input into the attribute prediction module for attribute label prediction, and the attribute prediction module does not output the attribute label corresponding to the classification label, the attribute label corresponding to the classification label is marked as no attribute label.

[0054] The image label acquisition method provided by the present application first acquires the global feature of the image to be classified, then inputs the global feature into the classification prediction module of the multi-label classification model, acquires the plurality of classification labels corresponding to the image to be classified and the feature map corresponding to each of the plurality of classification labels output by the classification prediction module, and finally inputs the feature map corresponding to each of the plurality of classification labels into the attribute prediction module of the multi-label classification model, to acquire the attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module. Through the above method, the global feature of the image to be classified is directly recognized by the multi-label classification model, and then the plurality of classification labels corresponding to the image to be classified and the feature map corresponding to each of the plurality of classification labels can be directly determined. After the feature map corresponding to each of the plurality of classification labels is determined, the attribute label corresponding to each classification label can be quickly determined, the process of acquiring the image label is simplified, and the speed of acquiring the image label is improved.

[0055] Please refer to Figure 4 The image label acquisition method provided by the embodiment of the present application is applied to the electronic device or the server as shown in Figure 1 or Figure 2 The method comprises the following steps:

[0056] Step S210: acquiring a training data set, wherein the training data set comprises a plurality of images and an image description sentence corresponding to each image.

[0057] In the embodiments of the present application, the image description language is textual content used to describe an image. The textual content used to describe an image can be a paragraph, a short sentence or a phrase, or a sentence, which is not limited herein. However, in the embodiments of the present application, each image in the training data set can correspond to one or more image description sentences. In the case where each image corresponds to multiple image description sentences, the multiple image description sentences can describe the image from different aspects. The different aspects can include objects in the image, a scene represented by the image, behaviors of objects in the image, information included in the image, and the like, which are not limited herein. For example, if image A corresponds to two image description sentences, one of the image description sentences can describe image A from the number and color of objects included in image A, and the other image description sentence can describe image A from the behaviors of the objects included in image A, which are not limited herein.

[0058] As a manner, the training data set can be a data set obtained from an existing database, or a data set constructed based on a preset rule, which is not limited herein. The preset rule is a rule for constructing a data set in advance, for example, a rule that one image corresponds to five image description sentences.

[0059] Optionally, the training data set can be obtained in response to a preset instruction. Specifically, the preset instruction is triggered when a specified application program is started, and the training data set is obtained. The specified application program can be an application program set in the electronic device for image acquisition.

[0060] Step S220: determining the attribute label and the classification label corresponding to each image based on the image description sentence corresponding to each image.

[0061] In the embodiments of the present application, the attribute label of an image can be the color and number of a target object included in the image, and the classification label of an image can be the category, behavior and scene of a target object included in the image.

[0062] Optionally, the image description sentence in the embodiments of the present application can comprehensively describe the image, and thus the comprehensive information of the image can be obtained from the image description sentence. Therefore, when performing multi-label classification on images in a large data set, in order to reduce the cost of manual labeling, it is not necessary to manually frame and label the instances in the image, and the label of the image can be directly determined from the image description sentence corresponding to the image.

[0063] As a manner, the image description sentence corresponding to the image can be input into a sentence parsing tool, the sentence components of the image description sentence are parsed by the sentence parsing tool, and a real word in the image description sentence corresponding to each image is taken as a classification label corresponding to each image. A modifier corresponding to the real word in the image description sentence corresponding to each image is taken as an attribute label corresponding to each image. The real word is a word with actual meaning in the image description sentence, and the real word can independently serve as a sentence component, that is, the real word is a word with lexical meaning and grammatical meaning. Optionally, the real word can be a word whose sentence component in the image description sentence is a subject or an object. The modifier corresponding to the real word can be understood as a word modifying the real word, and the modifier corresponding to the real word can be a word whose sentence component in the image description sentence is a determiner. For example, as shown in Figure 5 Figure 5 FIG. 1 is a schematic diagram of obtaining attribute labels and classification labels of images by a sentence parsing tool. In Figure 5 , the sentence parsing tool can parse the sentence components of each word in the image description sentence in the image description sentence, and then directly take the word whose sentence component in the image description sentence is a subject or an object as the classification label of the image, and take the word whose sentence component in the image description sentence is a determiner as the attribute label of the image.

[0064] Step S230: training the to-be-trained model based on the training data set and the attribute label and the classification label corresponding to each image in the training data set until a training end condition is met, to obtain the multi-label classification model.

[0065] In the embodiments of the present application, the training end condition can be that the loss value of the preset loss function reaches the preset loss value; the training end condition can also be that the number of iterations meets the preset number of iterations, and the like.

[0066] As a manner, the step of obtaining the multi-label classification model can further include: obtaining a preset loss function; training the to-be-trained model based on the preset loss function, the training data set, and the attribute label and the classification label corresponding to each image in the training data set until a training end condition is met, to obtain the multi-label classification model.

[0067] The preset loss function can be a pre-set cross-entropy loss function, and of course the preset loss function can also be other loss functions, which are not limited here.

[0068] When the to-be-trained model is trained by the loss function, the training data set, and the attribute label and the classification label corresponding to each image in the training data set, when it is detected that the loss value corresponding to the preset loss function meets the preset loss value, it is determined that the training end condition is met, and the multi-label classification model is obtained.

[0069] ​Step S240: obtaining a global feature of the image to be classified.

[0070] Step S250: inputting the global feature into a classification prediction module of the multi-label classification model, obtaining a plurality of classification labels corresponding to the image to be classified output by the classification prediction module, and a feature map corresponding to each of the plurality of classification labels.

[0071] Step S260: inputting the feature map corresponding to each of the plurality of classification labels into a weakly supervised localization module of the multi-label classification model, and obtaining a target object corresponding to each of the plurality of classification labels output by the weakly supervised localization module.

[0072] In the embodiments of the present application, the weakly supervised localization module divides the image to be classified into different instance segments, compares the instance segments in all instance segments containing the same classification label, and searches for the key instance therein. The instance segment refers to a part of the image region in the image to be classified. The key instance refers to the target object corresponding to the classification label.

[0073] As one way, when the image to be classified is divided into different instance segments, the image to be classified can be divided into different instance segments according to the target objects included in the image to be classified. For example, if the image to be classified includes target object 1, target object 2, target object 3 and target object 4, when the instance segment is divided, the image region where the target object 1 is located can be divided into an instance segment, the image region where the target object 2 is located can be divided into an instance segment, the image region where the target object 3 is located can be divided into an instance segment, and the image region where the target object 4 is located can be divided into an instance segment.

[0074] As another way, when the image to be classified is divided into different instance segments, the image to be classified can be divided into different instance segments according to the number of target objects included in the image to be classified. For example, if the image to be classified includes 5 target objects, when the instance segment is divided, each of the 5 target objects can be divided into an instance segment, or each of the 5 target objects can be divided into an instance segment, or 3 of the 5 target objects can be divided into an instance segment. The number of target objects included in the instance segment can be determined according to the type of the target object. If a first number of types of target objects are included in the plurality of target objects, the image to be classified is divided into a first number of instance segments; if a second number of types of target objects are included in the plurality of target objects, the image to be classified is divided into a second number of instance segments. Optionally, if a plurality of target objects are included in the instance segment, the target objects included in the instance segment can be of the same type or of different types.

[0075] Optionally, when the image to be classified is cut into different instance segments, the image to be classified can also be cut into different instance segments according to the preset image size. For example, if the size of the image to be classified is 8*8 and the preset image size is 2*2, when the image to be classified is cut into different instance segments, the image to be classified can be cut into 16 instance segments according to the size of 2*2.

[0076] Optionally, in the embodiment of the present application, the weakly supervised positioning module can compare the instance segments in all instance segments containing the same classification label through weakly supervised learning, and quickly locate the target object corresponding to the determined classification label in the image to be classified.

[0077] Step S270: inputting the target object corresponding to each of the plurality of classification labels into the attribute prediction module of the multi-label classification model, and obtaining the attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0078] As shown in Figure 6 , Figure 6 is a flowchart for implementing steps S240-S270. In Figure 6 , the image to be classified is input into the convolutional neural network to obtain the global feature corresponding to the image to be classified, and then the global feature of the image to be classified is input into the classification prediction module for classification label prediction to obtain the plurality of classification labels corresponding to the image to be classified and the feature map corresponding to each classification label. Then, the target object corresponding to each classification label in the feature map is located through weak supervision, and the located feature map is input into the attribute prediction module for attribute prediction to obtain the attribute label corresponding to each classification label.

[0079] In the process of predicting the attribute label corresponding to each classification label, the weakly supervised positioning module can be used to locate the target object corresponding to each classification label, so that the attribute prediction module can pay more attention to the features of each target object, and the prediction of the attribute label corresponding to each classification label can be more accurate.

[0080] The image label acquisition method provided by the present application directly identifies the global feature of the image to be classified through the multi-label classification model, and then directly determines the plurality of classification labels corresponding to the image to be classified and the feature map corresponding to each of the plurality of classification labels. After the feature map corresponding to each of the plurality of classification labels is determined, the attribute corresponding to each classification label can be quickly determined, which simplifies the process of acquiring the image label and improves the speed of acquiring the image label. Furthermore, the target object corresponding to each label in the image can be quickly located through the weakly supervised learning method, which is conducive to the subsequent processing of the target object corresponding to the specified label.

[0081] Referring to Figure 7 The embodiment of the present application provides an image label acquisition method, which is applied to an electronic device or a server as shown in Figure 1 Or Figure 2 The method comprises the following steps.

[0082] Step S310: taking an image in an image set as a to-be-classified image.

[0083] In the embodiment of the present application, in an actual application scenario, the image set can be a photo album, an image library or the like. As a kind of way, the images in the photo album or the image library can be taken as the to-be-classified images, and the images in the photo album or the image library are classified and predicted for the attribute label.

[0084] As a kind of way, any one of the images stored in the photo album or the image library can be taken as the to-be-classified image. Optionally, a plurality of images stored in the photo album or the image library can also be taken as the to-be-classified image. The number of to-be-classified images is specifically limited here.

[0085] Step S320: acquiring the global feature of the to-be-classified image.

[0086] Step S330: inputting the global feature into a classification prediction module of a multi-label classification model, acquiring a plurality of classification labels corresponding to the to-be-classified image and a feature map corresponding to each of the plurality of classification labels output by the classification prediction module.

[0087] Step S340: inputting the feature map corresponding to each of the plurality of classification labels into an attribute prediction module of the multi-label classification model, and acquiring the attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0088] Step S350: storing the plurality of classification labels of the to-be-classified image and the attribute label corresponding to each of the plurality of classification labels.

[0089] In the embodiment of the present application, when the plurality of classification labels of the to-be-classified image and the attribute label corresponding to each of the plurality of classification labels are obtained by the foregoing method, the plurality of classification labels of the to-be-classified image and the attribute label corresponding to each of the plurality of classification labels can be stored, so that in subsequent actual application scenarios, the corresponding image can be retrieved according to the stored classification label and the attribute label corresponding to the classification label.

[0090] As a kind of way, when the plurality of classification labels of the to-be-classified image and the attribute label corresponding to each of the plurality of classification labels are stored, a mapping relationship among the to-be-classified image, the classification label and the attribute label can be established, and the plurality of classification labels of the to-be-classified image and the attribute label corresponding to each of the plurality of classification labels are stored according to the mapping relationship.

[0091] Optionally, when storing the plurality of classification labels of the image to be classified and the respective attribute labels corresponding to each classification label, the mapping relationship among the image to be classified, the classification label and the attribute label can be stored in the local storage area. In order to prevent data loss, the mapping relationship among the image to be classified, the classification label and the attribute label can also be stored in the cloud server.

[0092] The image label acquisition method provided by the present application first takes an image in an image set as an image to be classified, acquires the global feature of the image to be classified, then inputs the global feature into a classification prediction module of a multi-label classification model, acquires the plurality of classification labels corresponding to the image to be classified and the respective feature maps corresponding to the plurality of classification labels output by the classification prediction module, inputs the respective feature maps corresponding to the plurality of classification labels into an attribute prediction module of the multi-label classification model, acquires the respective attribute labels corresponding to the plurality of classification labels output by the attribute prediction module, and finally stores the plurality of classification labels of the image to be classified and the respective attribute labels corresponding to the plurality of classification labels. Through the above method, the global feature of the image in the image set can be directly recognized by the multi-label classification model, and then the respective plurality of classification labels corresponding to each image in the image set and the respective feature maps corresponding to the plurality of classification labels can be directly determined. After the respective feature maps corresponding to the plurality of classification labels are determined, the respective attribute labels corresponding to each classification label can be quickly determined, the process of acquiring the image label is simplified, and the speed of acquiring the image label is improved.

[0093] Please refer to Figure 8 The image label acquisition method provided by the present application is applied to an electronic device or a server as shown in Figure 1 or Figure 2 The method comprises the following steps:

[0094] Step S410: acquiring an input preset label, wherein the preset label comprises at least one of a classification label and an attribute label.

[0095] In the present application, the preset label can be at least one of a classification label or an attribute label input by a user. When it is needed to acquire a plurality of images including the same classification label and / or the same attribute label from an album, the input preset label can be acquired.

[0096] As a kind of way, the preset label obtained can be one. Optionally, the preset label obtained can also be multiple, and the number of the preset label obtained can be determined according to actual search demand. Specifically, in the case where a large number of images need to be acquired, the number of the preset label obtained can be less; in the case where a certain image needs to be accurately acquired, the number of the preset label obtained can be more.

[0097] Step S420: comparing the preset label with the plurality of classification labels and the attribute label corresponding to each image in the album.

[0098] In the embodiments of the present application, each image in the album corresponds to at least one classification label, and each classification label can correspond to an attribute label.

[0099] After the input preset label is obtained, the preset label is compared with the plurality of classification labels and the attribute label corresponding to each image in the album respectively.

[0100] Step S430: obtaining a target image corresponding to the preset label from the album, and displaying the target image.

[0101] In the embodiments of the present application, the target image corresponding to the preset label is an image in the album having the same classification label and / or attribute label as the preset label.

[0102] As a manner, the image in the album having the same classification label and / or attribute label as the preset label can be one or more. If the target image corresponding to the preset label is more than one, when the target image is displayed, the plurality of target images can be displayed in layers, or the plurality of target images can be simultaneously displayed in a preset arrangement manner. When the plurality of target images are simultaneously displayed in the preset arrangement manner, the size of the plurality of target images can be reduced according to the size of the display screen, so that the display screen can simultaneously display the plurality of images. For example, as shown in FIG. 6, Figure 9 , the plurality of target images are displayed in layers; as shown in FIG. 7, Figure 9 , the plurality of target images are simultaneously displayed in the preset arrangement manner. Figure 10 Figure 10

[0103] As another manner, the displaying the target image comprises: displaying the target image, and highlighting the target object corresponding to the preset label in the target image.

[0104] In the target image, highlighting the target object corresponding to the preset label can be understood as deepening the color of the target object corresponding to the preset label when the target image is displayed, as shown in FIG. 8; or can be understood as highlighting the target object corresponding to the preset label by a preset size of a rectangular frame when the target image is displayed, as shown in FIG. 9; or can be understood as popping up a display frame to display the target object corresponding to the preset label when the target image is displayed, as shown in FIG. 10. Figure 11 Figure 12 Figure 13

[0105] ​​​​​The image label obtaining method provided in the application comprises the following steps: firstly, obtaining a preset label input; then, comparing the preset label with a plurality of classification labels and attribute labels corresponding to each of the images in an album, so that a target image corresponding to the preset label can be obtained from the album, and the target image is displayed. Through the above method, a multi-label image can be quickly and accurately retrieved based on a single label, and intelligent management of the album can be improved.

[0106] Referring to Figure 14 The image label obtaining device 500 provided in the embodiment of the application comprises the following:

[0107] The feature obtaining unit 510 is configured to obtain a global feature of a to-be-classified image.

[0108] The feature map obtaining unit 520 is configured to input the global feature into a classification prediction module of a multi-label classification model, and obtain a plurality of classification labels corresponding to the to-be-classified image and a feature map corresponding to each of the plurality of classification labels output by the classification prediction module.

[0109] The attribute obtaining unit 530 is configured to input the feature map corresponding to each of the plurality of classification labels into an attribute prediction module of the multi-label classification model, and obtain an attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0110] As a kind of mode, the attribute obtaining unit 530 is configured to input the target object corresponding to each of the plurality of classification labels into the attribute prediction module of the multi-label classification model, and obtain the attribute label corresponding to each of the plurality of classification labels output by the attribute prediction module.

[0111] Referring to Figure 15 The device 500 further comprises:

[0112] The model training unit 540 is configured to obtain a training data set, wherein the training data set comprises a plurality of images and an image description sentence corresponding to each image; determine an attribute label and a classification label corresponding to each image based on the image description sentence corresponding to each image; train a to-be-trained model based on the training data set and the attribute label and the classification label corresponding to each image in the training data set until a training end condition is met, to obtain the multi-label classification model.

[0113] As a kind of mode, the model training unit 540 is further configured to take a real word in the image description sentence corresponding to each image as the classification label corresponding to each image; and take a modifier corresponding to the real word in the image description sentence corresponding to each image as the attribute label corresponding to each image.

[0114] The image acquisition unit 550 is configured to acquire images in an album as images to be classified.

[0115] The storage unit 560 is configured to store a plurality of classification labels of the images to be classified and attribute labels corresponding to the plurality of classification labels respectively.

[0116] The positioning unit 570 is configured to input feature maps corresponding to the plurality of classification labels respectively into a weakly supervised positioning module of the multi-label classification model, and acquire target objects corresponding to the plurality of classification labels respectively output by the weakly supervised positioning module.

[0117] Referring to Figure 16 The image search device 600 provided by the embodiment of the present application comprises:

[0118] The label acquisition unit 610 is configured to acquire a preset label input, wherein the preset label comprises at least one of a classification label and an attribute label.

[0119] The comparison unit 620 is configured to compare the preset label with a plurality of classification labels and attribute labels corresponding to images in an album respectively.

[0120] The display unit 630 is configured to acquire target images corresponding to the preset label from the album and display the target images.

[0121] As a manner, the display unit 630 is specifically configured to display the target images and highlight target objects corresponding to the preset label in the target images.

[0122] It should be noted that the device embodiment in the present application corresponds to the foregoing method embodiment, and the specific principles of the device embodiment can be referred to the content in the foregoing method embodiment, which will not be described herein.

[0123] The following will be described in combination with Figure 17 An electronic device or a server provided by the present application will be described.

[0124] Referring to Figure 17 Based on the foregoing image label acquisition method and device, the embodiment of the present application further provides another electronic device or server 800 which can execute the foregoing image label acquisition method. The electronic device or server 800 comprises one or more (only one is shown in the figure) processors 802, a memory 804 and a network module 806 which are coupled to each other. The memory 804 stores programs which can execute the content in the foregoing embodiments, and the processor 802 can execute the programs stored in the memory 804.

[0125] The processor 802 may include one or more processing cores. The processor 802 connects to various parts of the electronic device or server 800 via various interfaces and lines, and performs various functions and processes data of the electronic device or server 800 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 804, and by calling data stored in the memory 804. Optionally, the processor 802 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 802 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 802 and may be implemented separately through a communication chip.

[0126] The memory 804 may include random access memory (RAM) or read-only memory (ROM). The memory 804 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 804 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created during the use of the electronic device or server 800 (such as phonebooks, audio and video data, chat log data, etc.).

[0127] The network module 806 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, thereby communicating with communication networks or other devices, such as audio playback devices. The network module 806 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, SIM cards, memory, etc. The network module 806 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices through wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. For example, the network module 806 can interact with base stations.

[0128] Please refer to Figure 18 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 900 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0129] The computer-readable storage medium 900 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 910 may, for example, be compressed in a suitable form.

[0130] This application provides an image label acquisition method, image search method, apparatus, electronic device, and storage medium. First, it acquires the global features of the image to be classified. Then, it inputs these global features into the classification prediction module of a multi-label classification model to acquire multiple classification labels corresponding to the image to be classified, as well as the feature maps corresponding to each of these labels. Finally, it inputs the feature maps corresponding to each of the multiple classification labels into the attribute prediction module of the multi-label classification model to acquire the attributes corresponding to each of the multiple classification labels, as output by the attribute prediction module. Through this method, the global features of the image to be classified are directly identified using a multi-label classification model, thereby directly determining the multiple classification labels corresponding to the image to be classified, as well as the feature maps corresponding to each of these labels. After determining the feature maps corresponding to each of the multiple classification labels, the attributes corresponding to each classification label can be quickly determined, simplifying the process of acquiring image labels and improving the speed of image label acquisition.

[0131] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for obtaining image tags, characterized in that, The method includes: Obtain the global features of the image to be classified; The global features are input into the classification prediction module of the multi-label classification model to obtain multiple classification labels corresponding to the image to be classified, as well as the feature maps corresponding to each of the multiple classification labels. The feature maps corresponding to each of the multiple classification labels are input into the weakly supervised localization module of the multi-label classification model to obtain the target objects corresponding to each of the multiple classification labels output by the weakly supervised localization module. The weakly supervised localization module divides the image to be classified into different instance segments, compares the instance segments in all instance segments containing the same classification label, and searches for key instances. The instance segment refers to a part of the image region in the image to be classified, and the key instance refers to the target object corresponding to the classification label. The target objects corresponding to each of the multiple classification labels are input into the attribute prediction module of the multi-label classification model to obtain the attribute labels corresponding to each of the multiple classification labels output by the attribute prediction module.

2. The method according to claim 1, characterized in that, Before obtaining the global features of the image to be classified, the following steps are also included: Obtain a training dataset, wherein the training dataset includes multiple images and image description statements corresponding to each image; Based on the image description statement corresponding to each image, determine the attribute label and category label corresponding to each image; Based on the training dataset, and the attribute labels and classification labels corresponding to each image in the training dataset, the model to be trained is trained until the training termination condition is met, thus obtaining the multi-label classification model.

3. The method according to claim 2, characterized in that, The process of determining the attribute label and category label for each image based on the image description statement corresponding to each image includes: Use the content words in the image description of each image as the category label for each image; Use the modifiers corresponding to the content words in the image description statement for each image as the attribute labels for each image.

4. The method according to claim 1, characterized in that, Before obtaining the global features of the image to be classified, the following steps are also included: Use the images in the image set as the images to be classified; After inputting the feature maps corresponding to each of the multiple classification labels into the attribute prediction module of the multi-label classification model, and obtaining the attribute labels corresponding to each of the multiple classification labels output by the attribute prediction module, the method further includes: Store multiple classification labels for the image to be classified, as well as the attribute labels corresponding to each of the multiple classification labels.

5. An image search method, characterized in that, The method includes: Obtain input preset tags, wherein the preset tags include at least one of category tags and attribute tags; The preset tags are compared with multiple category tags and attribute tags corresponding to each image in the album, wherein the multiple category tags and attribute tags corresponding to each image are obtained by the method described in claims 1-4; The target image corresponding to the preset tag is retrieved from the album and displayed.

6. The method according to claim 5, characterized in that, The process of displaying the target image includes: The target image is displayed, and the target object corresponding to the preset label is highlighted in the target image.

7. An image tag acquisition device, characterized in that, The device includes: The feature acquisition unit is used to acquire global features of the image to be classified. The feature map acquisition unit is used to input the global features into the classification prediction module of the multi-label classification model, and acquire multiple classification labels corresponding to the image to be classified output by the classification prediction module, as well as the feature maps corresponding to each of the multiple classification labels. The localization unit is used to input the feature maps corresponding to the multiple classification labels into the weakly supervised localization module of the multi-label classification model, and to obtain the target objects corresponding to the multiple classification labels output by the weakly supervised localization module. The weakly supervised localization module segments the image to be classified into different instance segments, compares the instance segments in all instance segments containing the same classification label, and searches for key instances. The instance segment refers to a part of the image region in the image to be classified, and the key instance refers to the target object corresponding to the classification label. The attribute acquisition unit is used to input the target objects corresponding to the multiple classification labels into the attribute prediction module of the multi-label classification model, and to acquire the attribute labels corresponding to the multiple classification labels output by the attribute prediction module.

8. An image search device, characterized in that, The device includes: A tag acquisition unit is used to acquire input preset tags, wherein the preset tags include at least one of category tags and attribute tags; A comparison unit is used to compare the preset label with multiple category labels and attribute labels corresponding to each image in the album, wherein the multiple category labels and attribute labels corresponding to each image are obtained by the method described in claims 1-4; The display unit is used to retrieve the target image corresponding to the preset tag from the album and display the target image.

9. An electronic device, characterized in that, It includes one or more processors; one or more programs are stored in memory and configured to be executed by the one or more processors according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein the program code, when executed by a processor, performs the method according to any one of claims 1-6.

Images