Image feature extraction model training method, image retrieval method and device

Abstract

The present disclosure provides a training method for an image feature extraction model, an image retrieval method and a device, and relates to the field of computer technology, and in particular to the field of big data, computer vision and deep learning technology. The specific implementation scheme is: obtaining a sample image set; extracting the visual semantic fusion features of the sample images in the sample image group; calculating the visual metric loss according to the visual semantic fusion features and the first label of the sample image group; calculating the semantic classification loss according to the visual semantic fusion features and the second label of each sample image in the sample image group; training the image feature extraction model based on the visual metric loss and the semantic classification loss. The visual semantic fusion features extracted by the model can represent both the visual information and the semantic information of the image. The visual semantic fusion features extracted by the model are used as the basis for retrieving similar images, so that the recalled similar images are visually similar to the retrieved images and semantically the same as the retrieved images.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of big data, computer vision and deep learning technology. Background Art

[0002] With the rapid development of deep learning, image retrieval technology based on deep learning has been widely used.

[0003] Through image retrieval technology, similar images of known images can be retrieved to obtain more relevant information. Summary of the invention

[0004] The present invention provides a training method for an image feature extraction model, an image retrieval method and a device.

[0005] According to one aspect of the present disclosure, a method for training an image feature extraction model is provided, comprising:

[0006] Acquire a sample image set, wherein the sample image set comprises a plurality of sample image groups having a first label, and each of the sample image groups comprises a plurality of sample images having a second label; the first label indicates whether the sample images included in the sample image group are of the same category; the second label indicates the category of the sample image; wherein the sample images of the same category have similar visual features and the same semantics;

[0007] Extracting visual semantic fusion features of sample images in the sample image group based on a first network structure included in the image feature extraction model;

[0008] Calculating visual metric loss according to the visual semantic fusion feature, the first label of the sample image group and a preset metric loss function;

[0009] Calculating semantic classification loss according to the visual semantic fusion feature, the second label of each sample image in the sample image group, and a preset classification loss function;

[0010] The image feature extraction model is trained based on the visual metric loss and the semantic classification loss.

[0011] According to another aspect of the present disclosure, there is provided an image retrieval method, comprising:

[0012] Get the target image;

[0013] A pre-trained image feature extraction model is used to extract the visual semantic fusion features of the target image, and similar images are retrieved from a retrieval library based on the visual semantic fusion features; the retrieval library stores images to be retrieved and retrieval indexes; wherein the retrieval index is constructed based on the visual semantic fusion features of the images to be retrieved; the visual semantic fusion features of the images to be retrieved are pre-extracted using the image feature extraction model.

[0014] According to another aspect of the present disclosure, there is provided a training device for an image feature extraction model, comprising:

[0015] An acquisition module is used to acquire a sample image set, wherein the sample image set includes a plurality of sample image groups with a first label, and each of the sample image groups includes a plurality of sample images with a second label; the first label indicates whether the sample images included in the sample image group are of the same category; the second label indicates the category of the sample image; wherein the sample images of the same category have similar visual features and the same semantics;

[0016] A feature extraction module, used for extracting visual semantic fusion features of sample images in the sample image group based on a first network structure included in the image feature extraction model;

[0017] A first calculation module, used for calculating a visual metric loss according to the visual semantic fusion feature, the first label of the sample image group and a preset metric loss function;

[0018] A second calculation module, used for calculating the semantic classification loss according to the visual semantic fusion feature, the second label of each sample image in the sample image group, and a preset classification loss function;

[0019] A training module is used to train the image feature extraction model based on the visual metric loss and the semantic classification loss.

[0020] According to another aspect of the present disclosure, there is provided an image retrieval device, comprising:

[0021] An acquisition module, used for acquiring a target image;

[0022] A retrieval module is used to extract the visual semantic fusion features of the target image using a pre-trained image feature extraction model, and retrieve similar images from a retrieval library based on the visual semantic fusion features; the retrieval library stores images to be retrieved and retrieval indexes; wherein the retrieval index is constructed based on the visual semantic fusion features of the images to be retrieved; the visual semantic fusion features of the images to be retrieved are pre-extracted using the image feature extraction model.

[0023] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0024] at least one processor; and

[0025] a memory communicatively connected to the at least one processor; wherein,

[0026] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a training method for an image feature extraction model or an image retrieval method.

[0027] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute an image feature extraction model training method or an image retrieval method.

[0028] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements a training method for an image feature extraction model or an image retrieval method when executed by a processor.

[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0031] Figure 1 A flowchart of a method for training an image feature extraction model provided in an embodiment of the present disclosure;

[0032] Figure 2 is a structural schematic diagram of an image feature extraction model provided by an embodiment of the present disclosure;

[0033] Figure 3 A schematic diagram of a flow chart of an image retrieval method provided by an embodiment of the present disclosure;

[0034] FIG4(a) is a schematic diagram of a target image provided by an embodiment of the present disclosure, FIG4(b) is a schematic diagram of a search result of an image search using a related technology, and FIG4(c) is a schematic diagram of an image search result provided by an embodiment of the present disclosure;

[0035] Figure 5 A schematic diagram of a search library construction method provided in an embodiment of the present disclosure;

[0036] Figure 6A schematic diagram of an image retrieval method provided by an embodiment of the present disclosure;

[0037] Figure 7 is a block diagram of a device for implementing the training method of the image feature extraction model of the embodiment of the present disclosure;

[0038] Figure 8 is a block diagram of a device for implementing the image retrieval method of an embodiment of the present disclosure;

[0039] Fig. 9 A block diagram of an electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0041] With the rapid development of mobile Internet, people are exposed to a large amount of data every day. Many people hope to filter out valuable information from the massive data. For example, they can retrieve similar images through the captured images to obtain information related to the images. This is the commonly used photo search. The core technology of photo search is the image search technology. Similar image retrieval is an important application scenario of the image search technology.

[0042] The goal of similar image retrieval is to return similar images that are not only similar to the retrieved images, but also to expect the subjects of the returned similar images to be consistent in category with the subjects of the retrieved images. In other words, it is necessary to ensure that the returned similar images are visually similar to the retrieved images, and also to ensure that the semantic information of the similar images and the retrieved images is the same.

[0043] At present, the relevant technical solutions for similar image retrieval are as follows:

[0044] First, the visual features of the query image are obtained through the visual retrieval model, and then the semantic features of the query image are obtained through the semantic retrieval model. The visual features and semantic features are spliced ​​to obtain the similar image retrieval features, and a preset number of images are retrieved from the gallery based on the similar image retrieval features.

[0045] In other words, it is necessary to train a visual retrieval model to extract the visual features of the image; and then train a semantic retrieval model to extract the semantic features of the image.

[0046] However, the above related technologies have the following shortcomings:

[0047] 1) Constructing training data consumes a lot of resources: that is, it is necessary to mine different types of targeted training data for the two models respectively.

[0048] 2) Model training consumes a lot of resources: that is, it is necessary to design the structures of the two models and the loss functions separately, and then train them separately, which not only requires a lot of manpower but also consumes training resources.

[0049] 3) The biggest shortcoming is that there is a risk of feature fusion. That is, after the visual retrieval model and the semantic retrieval model are trained separately, they can extract the visual features and semantic features of the image very well. However, when the spliced ​​features of these two features are used to retrieve similar images, the returned similar images are difficult to take into account both the visual and semantic aspects.

[0050] In addition, since the basis for retrieving similar images is the fusion of visual features and semantic features, for each image in the retrieval library, visual features and semantic features must be extracted separately, spliced ​​and saved. Since the image data in the retrieval library is currently huge, reaching 7 billion levels, it requires a lot of resources, such as offline GPU (graphics processing unit) resources, Hadoop (a distributed computing platform) computing resources and HDFS (a distributed file system) storage resources.

[0051] In order to solve the above technical problems, the present disclosure provides a training method for an image feature extraction model, including:

[0052] Acquire a sample image set, wherein the sample image set comprises a plurality of sample image groups having a first label, and each of the sample image groups comprises a plurality of sample images having a second label; the first label indicates whether the sample images included in the sample image group are of the same category; the second label indicates the category of the sample image; wherein the sample images of the same category have similar visual features and the same semantics;

[0053] Extracting visual semantic fusion features of sample images in the sample image group based on a first network structure included in the image feature extraction model;

[0054] Calculating visual metric loss according to the visual semantic fusion feature, the first label of the sample image group and a preset metric loss function;

[0055] Calculating semantic classification loss according to the visual semantic fusion feature, the second label of each sample image in the sample image group, and a preset classification loss function;

[0056] The image feature extraction model is trained based on the visual metric loss and the semantic classification loss.

[0057] It can be seen that in the disclosed embodiment, in the process of training the image feature extraction model, the visual semantic fusion features of the sample images in the sample image group are extracted. On the basis of the visual semantic fusion features, on the one hand, the visual metric loss is calculated using the loss function of metric learning, and the visual metric loss can characterize the visual discrimination of the model; on the other hand, the semantic classification loss is calculated using the loss function of the classification algorithm, and the semantic classification loss can characterize the semantic discrimination of the model. The image feature extraction model is trained by combining the visual metric loss and the semantic classification loss, and the visual discrimination and semantic discrimination of the model are continuously enhanced during the training process, that is, deep learning tasks at the visual level and the semantic level are performed simultaneously, which belongs to multi-task learning. Since both the visual metric loss and the semantic classification loss are calculated based on the visual semantic fusion features, after a period of training, the visual semantic fusion features extracted by the model can characterize both the visual information and the semantic information of the image.

[0058] Therefore, in the similar image retrieval process, the visual semantic fusion features extracted by the model are used as the basis for retrieving similar images, so that the recalled similar images take into account both visual and semantic aspects, that is, they are visually similar to the retrieved images and semantically identical to the retrieved images.

[0059] In addition, there is no need to separately train and maintain visual retrieval models and semantic retrieval models, nor is there any need to perform feature splicing, which greatly reduces the offline GPU resources, computing resources, and storage resources required during model training, feature extraction, and feature storage.

[0060] The following is a detailed introduction to the training method of the image feature extraction model provided in the embodiments of the present disclosure.

[0061] See also Figure 1 , Figure 1 A flow chart of a training method for an image feature extraction model provided in an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method may include the following steps:

[0062] S101: Obtain a sample image set, the sample image set comprising a plurality of sample image groups having a first label, each sample image group comprising a plurality of sample images having a second label; the first label indicating whether the sample images contained in the sample image group are of the same category; the second label indicating the category of the sample image; wherein sample images of the same category have similar visual features and the same semantics.

[0063] In the disclosed embodiment, a semi-supervised clustering algorithm may be used to construct training data for training an image feature extraction model.

[0064] Among them, the semi-supervised clustering algorithm is actually a classification algorithm, which is: obtaining random images from network traffic, extracting the features of random images, and using the topological relationship between the features to divide the images into different clusters, with one cluster corresponding to one entity category.

[0065] In order to ensure that the category coverage of the training data is wide enough, image data of different vertical categories can be obtained, such as covering vertical categories such as commodities and plants.

[0066] As an example, a semi-supervised clustering algorithm is used to cluster about 20 million sample images to obtain about 200,000 categories, and then a sample image set is constructed.

[0067] In the disclosed embodiment, sample images of the same or different categories may be combined to obtain a sample image group, and a first label may be marked for each sample image group according to whether the sample images included in the sample image group are of the same category.

[0068] As an example, the sample image group is a binary group. If the first label of the sample image group is "1", it means that the two sample images contained in the sample image group are of the same category; if the first label of the sample image group is "0", it means that the two sample images contained in the sample image group are of different categories.

[0069] In addition, a second label may be annotated for each sample image, where the second label indicates the category of the sample image. As an example, for sample images of the first category, ID1 is annotated as the second label; for sample images of the second category, ID2 is annotated as the second label, and so on.

[0070] In the disclosed embodiment, sample images of the same category have similar visual features and the same semantics.

[0071] Specifically, since the sample images of each category are obtained by clustering, the sample images of the same category must have similar visual features, that is, the image similarity is higher than a certain threshold. In addition, the semantics of the main part of the sample images of the same category are the same.

[0072] For example, the main parts of the sample images of a certain category are all pencils of model xx. It can also be understood that the semantics of these sample images are the same, that is, pencils of model xx.

[0073] S102: Extracting visual semantic fusion features of sample images in the sample image group based on a first network structure included in the image feature extraction model.

[0074] In the disclosed embodiment, a metric learning algorithm is used to improve the visual discrimination of the model, and a classification algorithm is used to improve the semantic discrimination of the model. The two are trained jointly and share the backbone (backbone network) of the model.

[0075] The first network structure included in the image feature extraction model is the backbone network, which is used to extract features of the sample image. Since the features extracted by the image feature extraction model in the disclosed embodiment can represent both the visual information and the semantic information of the image, they are recorded as visual semantic fusion features.

[0076] S103: Calculate the visual metric loss according to the visual semantic fusion feature, the first label of the sample image group and a preset metric loss function.

[0077] For ease of understanding, a brief introduction to deep metric learning is given. Deep metric learning is a method of metric learning, which aims to learn a mapping from original features to a low-dimensional dense vector space (called embedding space) so that objects of the same type are close in the embedding space, while objects of different types are far away in the embedding space.

[0078] There are many loss functions for deep metric learning, such as Contrastive loss, Tripletloss, Lifted Struct loss, etc. In the embodiments of the present disclosure, a metric loss function can be pre-selected.

[0079] In the embodiment of the present disclosure, the input of the image feature extraction model is a sample image of a sample image group, and the sample image group has a first label, which can be used to calculate the visual metric loss.

[0080] Specifically, the visual semantic fusion features of the sample images in each sample image group are mapped to the embedding space, and the features of the sample images in each sample image group in the embedding space and the first label of each sample image group are substituted into the metric loss function to obtain the loss value. Since the obtained loss value can represent the visual discrimination of the image feature extraction model, it is recorded as visual metric loss.

[0081] In one embodiment of the present disclosure, step S103 may specifically include:

[0082] S11: Processing the visual semantic fusion features based on the second network structure included in the image feature extraction model to obtain the visual features of each sample image in the sample image group.

[0083] S12: Calculate the visual metric loss according to the visual feature, the first label of the sample image group and a preset metric loss function.

[0084] Specifically, the second network structure may be a fully connected layer, which is used to process the visual semantic fusion features and extract visual features.

[0085] It is easy to understand that in the initial stage of model training, the second network structure does not have the ability to extract visual features from the visual semantic fusion features, but as the training progresses, the parameters in the second network structure are continuously adjusted and gradually acquire the above ability, thereby realizing the decoupling of visual features, and then calculating the visual metric loss based on the decoupled visual features.

[0086] The visual features extracted by the second network structure can be used as features of the embedding space. Therefore, the visual features of the sample images in each sample image group and the first label of each sample image group are substituted into the metric loss function to obtain the loss value. Since the obtained loss value can characterize the visual discrimination of the image feature extraction model, it is recorded as the visual metric loss.

[0087] It can be seen that in the disclosed embodiment, a second network structure is provided in the image feature extraction model for processing the visual semantic fusion features. As the training proceeds, the parameters in the second network structure are continuously adjusted, and the ability to extract visual features from the visual semantic fusion features is gradually acquired, thereby achieving the decoupling of visual features and avoiding the influence of semantically related features in the visual semantic fusion features on the visual metric loss.

[0088] S104: Calculate semantic classification loss according to the visual semantic fusion feature, the second label of each sample image in the sample image group, and a preset classification loss function.

[0089] The classification loss function may be a cross entropy loss function or other loss functions, without limitation.

[0090] Specifically, the category of each sample image is predicted according to the visual semantic fusion features, and the predicted category of each sample image and the second label of each sample image are substituted into the classification loss function to obtain the loss value. Since the obtained loss value can represent the semantic discrimination of the image feature extraction model, it is recorded as semantic metric loss.

[0091] In one embodiment of the present disclosure, step S103 may specifically include:

[0092] S21: Processing the visual semantic fusion features based on the third network structure included in the image feature extraction model to obtain the semantic features of each sample image in the sample image group, and determining the predicted category label of each sample image in the sample image group based on the semantic features.

[0093] S22: Calculate semantic classification loss based on the predicted category label, the second label of each sample image in the sample image group, and a preset classification loss function.

[0094] Specifically, the third network structure may be a fully connected layer, which is used to process visual semantic fusion features and extract semantic features.

[0095] It is easy to understand that in the initial stage of model training, the third network structure does not have the ability to extract semantic features from visual semantic fusion features, but as the training progresses, the parameters in the third network structure are continuously adjusted and gradually acquire the above-mentioned ability, thereby realizing the decoupling of semantic features, and then determining the predicted category label of each sample image based on the decoupled semantic features.

[0096] For example, the dimension of a semantic feature is equivalent to the number of category labels, and the predicted category label can be determined based on the values ​​of each dimension in the semantic feature.

[0097] Furthermore, the predicted category label of each sample image in the sample image group and the second label of each sample image in the sample image group are substituted into the preset classification loss function to obtain the loss value. Since the obtained loss value can characterize the semantic discrimination of the image feature extraction model, it is recorded as semantic classification loss.

[0098] It can be seen that in the disclosed embodiment, a third network structure is provided in the image feature extraction model for processing the visual semantic fusion features. As the training proceeds, the parameters in the third network structure are continuously adjusted, and the ability to extract semantic features from the visual semantic fusion features is gradually acquired, thereby achieving the decoupling of semantic features and avoiding the influence of vision-related features in the visual semantic fusion features on the semantic classification loss.

[0099] S105: Train the image feature extraction model based on visual metric loss and semantic classification loss.

[0100] In the disclosed embodiment, the image feature extraction model may be trained by combining the visual metric loss and the semantic classification loss.

[0101] For example, the visual metric loss and the semantic classification loss are weighted to obtain the total loss, and the parameters in the image feature extraction model are adjusted based on the total loss. The weights used in the weighting can be preset or continuously adjusted during the training process.

[0102] Therefore, as the training progresses, the visual metric loss and semantic classification loss gradually decrease. When the model converges or the loss value drops below the preset threshold, the model training is considered complete. The trained image feature extraction model has strong visual discrimination and semantic discrimination, and the visual metric loss and semantic classification loss are both calculated based on the visual semantic fusion features. Therefore, it can be considered that the visual semantic fusion features extracted by the trained image feature extraction model can well take into account the visual information and semantic information of the image.

[0103] It can be seen that in the disclosed embodiment, in the process of training the image feature extraction model, the visual semantic fusion features of the sample images in the sample image group are extracted. On the basis of the visual semantic fusion features, on the one hand, the visual metric loss is calculated using the loss function of metric learning, and the visual metric loss can characterize the visual discrimination of the model; on the other hand, the semantic classification loss is calculated using the loss function of the classification algorithm, and the semantic classification loss can characterize the semantic discrimination of the model. The image feature extraction model is trained by combining the visual metric loss and the semantic classification loss, and the visual discrimination and semantic discrimination of the model are continuously enhanced during the training process, that is, deep learning tasks at the visual level and the semantic level are performed simultaneously, which belongs to multi-task learning. Since both the visual metric loss and the semantic classification loss are calculated based on the visual semantic fusion features, after a period of training, the visual semantic fusion features extracted by the model can characterize both the visual information and the semantic information of the image.

[0104] Therefore, in the similar image retrieval process, the visual semantic fusion features extracted by the model are used as the basis for retrieving similar images, so that the recalled similar images take into account both visual and semantic aspects, that is, they are visually similar to the retrieved images and semantically identical to the retrieved images.

[0105] In addition, there is no need to separately train and maintain visual retrieval models and semantic retrieval models, nor is there any need to perform feature splicing, which greatly reduces the offline GPU resources, computing resources, and storage resources required during model training, feature extraction, and feature storage.

[0106] For ease of understanding, the following Figure 2 The image feature extraction model provided by the embodiment of the present disclosure is further explained.

[0107] See also Figure 2 , Figure 2 Schematic diagram of a structure of an image feature extraction model provided by an embodiment of the present disclosure. Figure 2 As shown, the image feature extraction model includes a first network structure, namely a backbone for extracting features, which may include resnet50 and two fully connected layers ( Figure 2 denoted as FC1 and FC2 respectively), where resnet50 can be pre-trained using imagenet (an image recognition database).

[0108] The input training data is a batch of sample image groups, and each sample image group has a first label, indicating whether the sample images contained in the sample image group are of the same category; each sample image has a second label, indicating the category of the sample image.

[0109] The first network structure in the image feature extraction model extracts the visual semantic fusion features of the sample image, and then is divided into two paths to calculate the visual metric loss and semantic classification loss respectively.

[0110] Figure 2 The second network structure shown processes the visual semantic fusion features, extracts visual features from them, and calculates the visual metric loss based on the visual features; Figure 2 The third network structure shown processes the visual semantic fusion features, extracts semantic features from them, and calculates the semantic classification loss based on the semantic features. Then, the visual metric loss and the semantic classification loss are weighted to obtain the total loss, and the parameters in the image feature extraction model are adjusted according to the total loss.

[0111] As an example, during training, the warmup method can be used to optimize the learning rate, and the learning rate can be set to decrease step by step after a period of time. The weight of the semantic classification loss remains unchanged, and the warmup method is used to optimize the weight of the visual metric loss.

[0112] In one embodiment of the present disclosure, in order to allow the visual metric loss and the semantic classification loss to be distributed in the same feature space, the features can be processed through a BNNeck network structure before calculating the semantic classification loss. That is, a BNNeck network structure is added after the third network structure so that the two losses can be weighted to achieve the effect of joint training.

[0113] See also Figure 3 , Figure 3 A flowchart of an image retrieval method provided by an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the method may include the following steps:

[0114] S301: Acquire a target image.

[0115] The target image can be any image input by the user. The purpose of image retrieval is to recall similar images of the target image from the retrieval library, satisfying similar visual features and the same semantics.

[0116] S302: Using a pre-trained image feature extraction model to extract visual semantic fusion features of the target image, and retrieving similar images from a retrieval library based on the visual semantic fusion features; the retrieval library stores images to be retrieved and retrieval indexes; wherein the retrieval index is constructed based on the visual semantic fusion features of the images to be retrieved; the visual semantic fusion features of the images to be retrieved are pre-extracted using an image feature extraction model.

[0117] In the embodiment of the present disclosure, it is possible to adopt Figure 1The steps shown train the image feature extraction model, thereby using the trained image feature extraction model to extract the visual semantic fusion features of the target image, and retrieving similar images from the retrieval library based on the visual semantic fusion features.

[0118] Among them, a large number of images to be retrieved are stored in the retrieval library, and the image feature extraction model is used in advance to extract the visual semantic fusion features of these images to be retrieved, and an index is constructed based on the visual semantic fusion features.

[0119] The process of building an index based on features can refer to related technologies, for example, using an ANN (Approximate Nearest Neighbor) retrieval method to build an index for a retrieval library.

[0120] It can be seen that in the disclosed embodiment, the image feature extraction model is trained by combining the visual metric loss and the semantic classification loss, and the visual discrimination and semantic discrimination of the model are continuously enhanced during the training process. Since both the visual metric loss and the semantic classification loss are calculated based on the visual semantic fusion features, after a period of training, the visual semantic fusion features extracted by the model can represent both the visual information and the semantic information of the image.

[0121] Therefore, in the similar image retrieval process, the visual semantic fusion features extracted by the model are used as the basis for retrieving similar images, so that the recalled similar images take into account both visual and semantic aspects, that is, they are visually similar to the retrieved images and semantically identical to the retrieved images.

[0122] As an example, refer to Figures 4(a), 4(b) and 4(c), where Figure 4(a) is a schematic diagram of a target image provided by an embodiment of the present disclosure, Figure 4(b) is a schematic diagram of a retrieval result of image retrieval using related technologies, and Figure 4(c) is a schematic diagram of an image retrieval result provided by an embodiment of the present disclosure.

[0123] It can be seen that the main part of the target image is "guitar tuner", that is, the semantics of the main part of the target image is "guitar tuner"; Figure 4(b) shows the retrieval results of image retrieval using related technologies. Although the retrieved images are similar to the target images in visual features, most of them do not meet the requirements of semantic similarity. For example, the main parts of the retrieved images are: "intercom" and "air conditioner remote control". Using the training method and image retrieval method of the image feature extraction model provided by the embodiment of the present disclosure, the image retrieval results are shown in Figure 4(c). It can be seen that the retrieved images are similar to the target images in visual features, and the retrieved images meet the requirements of semantic similarity, both of which are "guitar tuner".

[0124] In the disclosed embodiment, in order to further improve the image retrieval efficiency, principal component analysis can be performed based on a certain number of image pairs. For example, 30,000 to 40,000 image pairs are randomly selected from a retrieval library of related technologies, and image features are extracted. The image features of these image pairs are used as training data, and a mapping matrix is ​​trained using a principal component analysis algorithm. The mapping matrix processes the image features to achieve dimensionality reduction, thereby reducing the complexity of constructing the index in the retrieval library and improving image retrieval efficiency.

[0125] Among them, the specific method of principal component analysis can be found in related technologies and will not be described in detail.

[0126] Correspondingly, for a large number of images to be retrieved in the retrieval library of the related technology, after extracting the visual semantic fusion features, feature mapping is performed based on the mapping matrix, and then an index is established according to the mapped features.

[0127] For details, please refer to Figure 5 , Figure 5 A schematic diagram of a search library construction method provided in an embodiment of the present disclosure, such as Figure 5 As shown, there are tens of billions of images to be retrieved in the retrieval library of the related technology. Visual semantic fusion features are extracted through a trained image feature extraction model, and then feature mapping is performed based on the mapping matrix obtained in the principal component analysis process. Then, an index is established based on the mapped features. Finally, the retrieval library stores the index and a large number of images to be retrieved.

[0128] In one embodiment of the present disclosure, the retrieval library includes a time-sensitive image library, which is constructed based on time-sensitive images and visual semantic fusion features of the time-sensitive images.

[0129] Specifically, the retrieval library of the related technology is a fixed database, but in real life, a huge amount of image data is generated on the Internet every day, and these data are not included; therefore, if users search for images with high timeliness, it is difficult to return similar images from the fixed retrieval library, which cannot meet the needs of users and seriously affects the user experience.

[0130] Considering the above shortcomings, in the embodiment of the present disclosure, time-sensitive images are continuously collected, and then a time-sensitive image library is established. The specific process of establishing a search library based on images can be found in Figure 5 , no more details.

[0131] As an example, daily time-efficiency data is acquired, and then time-efficiency images are collected, and the time-efficiency images are kept updated, for example, images collected in the last 50 days are kept as time-efficiency images.

[0132] Therefore, in the process of image retrieval, similar images are retrieved from the retrieval library including the time-sensitive image library.

[0133] For details, please refer to Figure 6 , Figure 6 A schematic diagram of an image retrieval method provided by an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the target image obtains visual semantic fusion features through the trained image feature extraction model, and then feature mapping is performed based on the mapping matrix obtained in the principal component analysis process. Based on the mapped features, similar images are retrieved in the retrieval library containing the time-sensitive image library, and a preset number of similar images are returned.

[0134] It can be seen that in the embodiment of the present disclosure, a time-sensitive retrieval library is established based on time-sensitive images, thereby being able to provide time-sensitive retrieval services. If a user searches for images with strong timeliness, similar images can also be returned, thereby meeting user needs and improving user experience.

[0135] See also Figure 7 , Figure 7 The device for implementing the training method of the image feature extraction model of the embodiment of the present disclosure is a block diagram, comprising the following modules:

[0136] The acquisition module 701 is used to acquire a sample image set, wherein the sample image set includes a plurality of sample image groups with a first label, and each of the sample image groups includes a plurality of sample images with a second label; the first label indicates whether the sample images included in the sample image group are of the same category; the second label indicates the category of the sample image; wherein the sample images of the same category have similar visual features and the same semantics;

[0137] A feature extraction module 702, configured to extract visual semantic fusion features of sample images in the sample image group based on a first network structure included in the image feature extraction model;

[0138] A first calculation module 703, configured to calculate a visual metric loss according to the visual semantic fusion feature, the first label of the sample image group, and a preset metric loss function;

[0139] A second calculation module 704 is used to calculate the semantic classification loss according to the visual semantic fusion feature, the second label of each sample image in the sample image group, and a preset classification loss function;

[0140] The training module 705 is used to train the image feature extraction model based on the visual metric loss and the semantic classification loss.

[0141] In one embodiment of the present disclosure, the first calculation module 703 is specifically used to:

[0142] Processing the visual semantic fusion feature based on the second network structure included in the image feature extraction model to obtain the visual feature of each sample image in the sample image group;

[0143] A visual metric loss is calculated according to the visual feature, the first label of the sample image group, and a preset metric loss function.

[0144] In one embodiment of the present disclosure, the second calculation module 704 is specifically configured to:

[0145] Processing the visual semantic fusion feature based on the third network structure included in the image feature extraction model to obtain the semantic feature of each sample image in the sample image group, and determining the predicted category label of each sample image in the sample image group based on the semantic feature;

[0146] The semantic classification loss is calculated based on the predicted category label, the second label of each sample image in the sample image group, and a preset classification loss function.

[0147] It can be seen that in the disclosed embodiment, in the process of training the image feature extraction model, the visual semantic fusion features of the sample images in the sample image group are extracted. On the basis of the visual semantic fusion features, on the one hand, the visual metric loss is calculated using the loss function of metric learning, and the visual metric loss can characterize the visual discrimination of the model; on the other hand, the semantic classification loss is calculated using the loss function of the classification algorithm, and the semantic classification loss can characterize the semantic discrimination of the model. The image feature extraction model is trained by combining the visual metric loss and the semantic classification loss, and the visual discrimination and semantic discrimination of the model are continuously enhanced during the training process, that is, deep learning tasks at the visual level and the semantic level are performed simultaneously, which belongs to multi-task learning. Since both the visual metric loss and the semantic classification loss are calculated based on the visual semantic fusion features, after a period of training, the visual semantic fusion features extracted by the model can characterize both the visual information and the semantic information of the image.

[0148] Therefore, in the similar image retrieval process, the visual semantic fusion features extracted by the model are used as the basis for retrieving similar images, so that the recalled similar images take into account both visual and semantic aspects, that is, they are visually similar to the retrieved images and semantically identical to the retrieved images.

[0149] In addition, there is no need to separately train and maintain visual retrieval models and semantic retrieval models, nor is there any need to perform feature splicing, which greatly reduces the offline GPU resources, computing resources, and storage resources required during model training, feature extraction, and feature storage.

[0150] See also Figure 8 , Figure 8 The device for implementing the image retrieval method of the embodiment of the present disclosure is a block diagram, comprising the following modules:

[0151] An acquisition module 801 is used to acquire a target image;

[0152] The retrieval module 802 is used to extract the visual semantic fusion features of the target image using a pre-trained image feature extraction model, and retrieve similar images from a retrieval library based on the visual semantic fusion features; the retrieval library stores images to be retrieved and retrieval indexes; wherein the retrieval index is constructed based on the visual semantic fusion features of the images to be retrieved; the visual semantic fusion features of the images to be retrieved are pre-extracted using the image feature extraction model.

[0153] It can be seen that in the disclosed embodiment, the image feature extraction model is trained by combining the visual metric loss and the semantic classification loss, and the visual discrimination and semantic discrimination of the model are continuously enhanced during the training process. Since both the visual metric loss and the semantic classification loss are calculated based on the visual semantic fusion features, after a period of training, the visual semantic fusion features extracted by the model can represent both the visual information and the semantic information of the image.

[0154] Therefore, in the similar image retrieval process, the visual semantic fusion features extracted by the model are used as the basis for retrieving similar images, so that the recalled similar images take into account both visual and semantic aspects, that is, they are visually similar to the retrieved images and semantically identical to the retrieved images.

[0155] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0156] The present disclosure provides an electronic device, comprising:

[0157] at least one processor; and

[0158] a memory communicatively connected to the at least one processor; wherein,

[0159] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a training method for an image feature extraction model or an image retrieval method.

[0160] The present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute a training method for an image feature extraction model or an image retrieval method.

[0161] The present disclosure provides a computer program product, including a computer program, which implements a training method for an image feature extraction model or an image retrieval method when executed by a processor.

[0162] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0163] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0164] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0165] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the training method of the image feature extraction model or the image retrieval method. For example, in some embodiments, the training method of the image feature extraction model or the image retrieval method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method of the image feature extraction model or the image retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the training method of the image feature extraction model or the image retrieval method in any other appropriate manner (for example, by means of firmware).

[0166] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0168] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0170] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0171] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0172] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0173] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for an image feature extraction model, comprising: A sample image set is obtained, wherein the sample image set includes a plurality of sample image groups with a first label, and each of the sample image groups includes a plurality of sample images with a second label; the first label indicates whether the sample images included in the sample image group are of the same category; the second label indicates the category of the sample image; wherein sample images of the same category have similar visual features and the same semantics; obtaining the sample image set includes: obtaining random images of different vertical categories, extracting features of the random images, clustering the random images according to the topological relationship between the features of the random images to obtain a plurality of clusters, one cluster corresponding to one category, and combining random images of the same or different categories to obtain a plurality of sample image groups; Extracting visual semantic fusion features of sample images in the sample image group based on a first network structure included in the image feature extraction model; Calculating visual metric loss according to the visual semantic fusion feature, the first label of the sample image group and a preset metric loss function, including: mapping the visual semantic fusion feature of the sample image in the sample image group to an embedding space, and calculating the visual metric loss based on the features of the sample image in the sample image group in the embedding space, the first label of the sample image group and a preset metric loss function; Calculating semantic classification loss according to the visual semantic fusion feature, the second label of each sample image in the sample image group, and a preset classification loss function, including: predicting the category of each sample image in the sample image group according to the visual semantic fusion feature, and calculating semantic classification loss based on the predicted category of each sample image, the second label of each sample image in the sample image group, and a preset classification loss function; The image feature extraction model is trained based on the visual metric loss and the semantic classification loss.

2. The method according to claim 1, wherein: The step of calculating the visual metric loss according to the visual semantic fusion feature, the first label of the sample image group and a preset metric loss function comprises: Processing the visual semantic fusion feature based on the second network structure included in the image feature extraction model to obtain the visual feature of each sample image in the sample image group; The visual metric loss is calculated according to the visual feature, the first label of the sample image group and a preset metric loss function.

3. The method according to claim 1, wherein: The step of calculating the semantic classification loss according to the visual semantic fusion feature, the second label of each sample image in the sample image group, and a preset classification loss function comprises: Processing the visual semantic fusion feature based on the third network structure included in the image feature extraction model to obtain the semantic feature of each sample image in the sample image group, and determining the predicted category label of each sample image in the sample image group based on the semantic feature; The semantic classification loss is calculated based on the predicted category label, the second label of each sample image in the sample image group, and a preset classification loss function.

4. An image retrieval method, comprising: Get the target image; A pre-trained image feature extraction model is used to extract visual semantic fusion features of the target image, and similar images are retrieved from a retrieval library based on the visual semantic fusion features; The retrieval library stores images to be retrieved and retrieval indexes; wherein the retrieval index is constructed based on the mapped features obtained by processing the visual semantic fusion features of the images to be retrieved based on the mapping matrix obtained by principal component analysis; The visual semantic fusion features of the image to be retrieved are extracted in advance using the image feature extraction model; the pre-trained image feature extraction model is trained based on the method described in any one of claims 1 to 3; The retrieval library includes a time-effect image library, and the time-effect image library is constructed based on time-effect images and visual semantic fusion features of the time-effect images.

5. A training device for an image feature extraction model, comprising: An acquisition module is used to acquire a sample image set, wherein the sample image set includes multiple sample image groups with a first label, and each of the sample image groups includes multiple sample images with a second label; the first label indicates whether the sample images included in the sample image group are of the same category; the second label indicates the category of the sample image; wherein sample images of the same category have similar visual features and the same semantics; acquiring the sample image set includes: acquiring random images of different vertical categories, extracting features of the random images, clustering the random images according to the topological relationship between the features of the random images to obtain multiple clusters, one cluster corresponding to one category, and combining random images of the same or different categories to obtain multiple sample image groups; A feature extraction module, used for extracting visual semantic fusion features of sample images in the sample image group based on a first network structure included in the image feature extraction model; A first calculation module, used for calculating a visual metric loss according to the visual semantic fusion feature, the first label of the sample image group and a preset metric loss function; A second calculation module, used for calculating the semantic classification loss according to the visual semantic fusion feature, the second label of each sample image in the sample image group, and a preset classification loss function; A training module, used for training the image feature extraction model based on the visual metric loss and the semantic classification loss; The first calculation module is specifically used to map the visual semantic fusion features of the sample images in the sample image group to the embedding space, and calculate the visual metric loss based on the features of the sample images in the sample image group in the embedding space, the first label of the sample image group and the preset metric loss function; The second calculation module is specifically used to predict the category of each sample image in the sample image group according to the visual semantic fusion feature, and calculate the semantic classification loss based on the predicted category of each sample image, the second label of each sample image in the sample image group, and a preset classification loss function.

6. The device according to claim 5, wherein: The first calculation module is specifically used for: Processing the visual semantic fusion feature based on the second network structure included in the image feature extraction model to obtain the visual feature of each sample image in the sample image group; A visual metric loss is calculated according to the visual feature, the first label of the sample image group, and a preset metric loss function.

7. The device according to claim 5, wherein: The second calculation module is specifically used for: Processing the visual semantic fusion feature based on the third network structure included in the image feature extraction model to obtain the semantic feature of each sample image in the sample image group, and determining the predicted category label of each sample image in the sample image group based on the semantic feature; The semantic classification loss is calculated based on the predicted category label, the second label of each sample image in the sample image group, and a preset classification loss function.

8. An image retrieval device, comprising: An acquisition module, used for acquiring a target image; A retrieval module, used to extract visual semantic fusion features of the target image using a pre-trained image feature extraction model, and retrieve similar images from a retrieval library based on the visual semantic fusion features; The retrieval library stores images to be retrieved and retrieval indexes; wherein the retrieval index is constructed based on the mapped features obtained by processing the visual semantic fusion features of the images to be retrieved based on the mapping matrix obtained by principal component analysis; The visual semantic fusion features of the image to be retrieved are extracted in advance using the image feature extraction model; the pre-trained image feature extraction model is trained based on the method described in any one of claims 1 to 3; The retrieval library includes a time-effect image library, and the time-effect image library is constructed based on time-effect images and visual semantic fusion features of the time-effect images.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-4.

11. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images