Method and apparatus for determining similar images

Abstract

The embodiment of the present application discloses a method and device for determining similar images. A specific implementation of the method includes: inputting a target image into a feature extraction network in a trained image similarity evaluation model to extract image features and obtain a feature vector of the target image; based on the feature vector of the target image and a feature matrix of a candidate image set obtained in advance, determining similar images of the target image from a candidate image set; wherein the feature matrix of the candidate image set is constructed by the feature vectors of each candidate image in the candidate image set, and the feature vectors of the candidate images are obtained by extracting features from the candidate images based on the feature extraction network in the trained image similarity evaluation model. This implementation improves the accuracy and efficiency of similar image search.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, specifically to the field of image processing technology, and in particular to a method and apparatus for determining similar images. Background Art

[0002] With the development of network technology, more and more Internet data is generated. Search engines can collect and organize massive amounts of Internet data. Users can enter key description information through search engines based on crawler technology to obtain data content. Commonly used key description information is text keywords, which can be used for matching during search.

[0003] In some platforms, data content is described in a variety of ways. For example, data content in e-commerce platforms is mostly based on pictures combined with text descriptions. The amount of search results obtained by using traditional text keyword retrieval methods on these platforms is large, and manual secondary screening is required. In addition, in some scenarios, users want to view similar content to known data content. They can use the background search engine to crawl the data content on the network in advance and search for similar content based on the index constructed based on the text description information of the data content. This content search method is highly dependent on the data index mode of the background search engine. Summary of the invention

[0004] The embodiments of the present application provide a method and device for determining similar pictures.

[0005] In a first aspect, an embodiment of the present application provides a method for determining similar pictures, comprising: inputting a target picture into a feature extraction network in a trained image similarity evaluation model to perform image feature extraction to obtain a feature vector of the target picture; based on the feature vector of the target picture and a feature matrix of a candidate picture set obtained in advance, determining similar pictures of the target picture from a candidate picture set; wherein the feature matrix of the candidate picture set is constructed by feature vectors of each candidate picture in the candidate picture set, and the feature vectors of the candidate pictures are obtained by performing feature extraction on the candidate pictures based on the feature extraction network in the trained image similarity evaluation model.

[0006] In some embodiments, the feature vector of the candidate image is a normalized row feature vector, and the feature vector of the target image is a normalized column feature vector; in the feature matrix of the candidate image set, the feature vector of each candidate image is a row vector in the feature matrix; based on the feature vector of the target image and the feature matrix of the candidate image set obtained in advance, similar images of the target image are determined from the candidate image set, including: determining the maximum value of the elements in the result column vector obtained by multiplying the feature matrix with the feature vector of the target image, and determining that the candidate image represented by the feature vector corresponding to the position of the maximum value in the result column vector in the feature matrix is ​​a similar image of the target image.

[0007] In some embodiments, the feature vector of the candidate image is a normalized column feature vector, and the feature vector of the target image is a normalized row feature vector; in the feature matrix of the candidate image set, the feature vector of each candidate image is a column vector in the feature matrix; based on the feature vector of the target image and the feature matrix of the candidate image set obtained in advance, similar images of the target image are determined from the candidate image set, including: determining the maximum value of the elements in the resultant row vector obtained by multiplying the feature vector of the target image and the feature matrix, and determining that the candidate image represented by the feature vector corresponding to the position of the maximum value in the resultant row vector in the feature matrix is ​​a similar image of the target image.

[0008] In some embodiments, the above-mentioned image similarity evaluation model includes a similarity calculation network and two feature extraction networks with shared weights; the above-mentioned method also includes: training a set of sample image pairs constructed from candidate images to obtain a trained image similarity evaluation model.

[0009] In some embodiments, the above-mentioned sample picture pair set includes sample picture pairs formed by candidate pictures and similar attribute annotation information of the sample picture pairs; the above-mentioned trained image similarity evaluation model obtained by training the sample picture pair set constructed by the candidate pictures includes: inputting two sample pictures in the sample picture pair into two feature extraction networks in the image similarity evaluation model to be trained respectively, to obtain feature vectors of the two sample pictures in the sample picture pair; based on the feature vectors of the two sample pictures in the sample picture pair, using the similarity calculation network in the image similarity evaluation model to be trained to calculate the similarity of the two sample pictures in the sample picture pair; based on a preset loss function, using the back propagation method to iteratively adjust the weights of the feature extraction network in the image similarity evaluation model to be trained, so that the value of the loss function satisfies the preset convergence condition, wherein the value of the loss function is used to characterize the difference between the similarity of the sample picture pair calculated by the image similarity evaluation model to be trained and the annotation information of the similar attributes of the corresponding sample picture pair.

[0010] In some embodiments, the above method also includes: obtaining user feedback information on the similarity evaluation between the target image and the similar image of the target image; determining the similar attribute labeling information between the target image and the similar image of the target image based on the user feedback information, and adding the target image and the similar image of the target image as a sample image pair to the sample image pair set.

[0011] In a second aspect, an embodiment of the present application provides a device for determining similar pictures, comprising: an extraction unit, configured to input a target picture into a feature extraction network in a trained image similarity evaluation model to perform image feature extraction to obtain a feature vector of the target picture; a determination unit, configured to determine similar pictures of the target picture from a candidate picture set based on the feature vector of the target picture and a feature matrix of a pre-acquired candidate picture set; wherein the feature matrix of the candidate picture set is constructed by feature vectors of each candidate picture in the candidate picture set, and the feature vectors of the candidate pictures are obtained by extracting features of the candidate pictures based on the feature extraction network in the trained image similarity evaluation model.

[0012] In some embodiments, the feature vector of the candidate image is a normalized row feature vector, and the feature vector of the target image is a normalized column feature vector; in the feature matrix of the candidate image set, the feature vector of each candidate image is a row vector in the feature matrix; the determination unit is further configured to determine similar images of the target image from the candidate image set based on the feature vector of the target image and the feature matrix of the candidate image set obtained in advance in the following manner: determine the maximum value of the elements in the result column vector obtained by multiplying the feature matrix by the feature vector of the target image, and determine that the candidate image represented by the feature vector corresponding to the position of the maximum value in the result column vector in the feature matrix is ​​a similar image of the target image.

[0013] In some embodiments, the feature vector of the candidate image is a normalized column feature vector, and the feature vector of the target image is a normalized row feature vector; in the feature matrix of the candidate image set, the feature vector of each candidate image is a column vector in the feature matrix; the determination unit is further configured to determine similar images of the target image from the candidate image set based on the feature vector of the target image and the feature matrix of the candidate image set obtained in advance in the following manner: determine the maximum value of the elements in the resultant row vector obtained by multiplying the feature vector of the target image by the feature matrix, and determine that the candidate image represented by the feature vector corresponding to the position of the maximum value in the resultant row vector in the feature matrix is ​​a similar image of the target image.

[0014] In some embodiments, the above-mentioned image similarity evaluation model includes a similarity calculation network and two feature extraction networks with shared weights; the above-mentioned device also includes: a training unit, configured to train a set of sample image pairs constructed by candidate images to obtain a trained image similarity evaluation model.

[0015] In some embodiments, the above-mentioned sample picture pair set includes a sample picture pair formed by candidate pictures and similar attribute annotation information of the sample picture pair; the above-mentioned training unit is further configured to train and obtain a trained image similarity evaluation model in the following manner: two sample pictures in the sample picture pair are respectively input into two feature extraction networks in the image similarity evaluation model to be trained to obtain feature vectors of the two sample pictures in the sample picture pair; based on the feature vectors of the two sample pictures in the sample picture pair, the similarity of the two sample pictures in the sample picture pair is calculated using the similarity calculation network in the image similarity evaluation model to be trained; based on a preset loss function, the weights of the feature extraction network in the image similarity evaluation model to be trained are iteratively adjusted using a back propagation method so that the value of the loss function satisfies a preset convergence condition, wherein the value of the loss function is used to characterize the difference between the similarity of the sample picture pair calculated by the image similarity evaluation model to be trained and the annotation information of the similar attributes of the corresponding sample picture pair.

[0016] In some embodiments, the above-mentioned device also includes an updating unit, which is configured to: obtain user feedback information on the similarity evaluation between the target image and the similar image of the target image; determine the similar attribute labeling information between the target image and the similar image of the target image based on the user feedback information, and add the target image and the similar image of the target image as a sample image pair to the sample image pair set.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining similar images provided in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for determining similar images provided in the first aspect is implemented.

[0019] The method and device for determining similar pictures in the above-mentioned embodiments of the present application obtain a feature vector of the target picture by inputting the target picture into a feature extraction network in a trained image similarity evaluation model to extract image features; based on the feature vector of the target picture and a feature matrix of a candidate picture set obtained in advance, similar pictures of the target picture are determined from the candidate picture set; wherein the feature matrix of the candidate picture set is constructed by the feature vectors of each candidate picture in the candidate picture set, and the feature vectors of the candidate pictures are obtained by extracting features of the candidate pictures based on the feature extraction network in the trained image similarity evaluation model, thereby realizing fast and accurate positioning of similar pictures and improving the efficiency and accuracy of searching for similar pictures. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0021] Figure 1 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0022] Figure 2 is a flow chart of an embodiment of a method for determining similar pictures according to the present application;

[0023] Figure 3 is a flowchart of another embodiment of a method for determining similar pictures according to the present application;

[0024] Figure 4 is based on Figure 3 A schematic diagram of the structure of an image similarity evaluation model in a method for determining similar images shown;

[0025] Figure 5 is a structural schematic diagram of an embodiment of a device for determining similar pictures of the present application;

[0026] Figure 6 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0027] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0028] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] Figure 1 An exemplary system architecture 100 is shown to which the method for determining similar pictures or the apparatus for determining similar pictures of the present application can be applied.

[0030] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0031] The user 110 can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 can be installed with various applications with information search functions, such as e-commerce applications, review applications, social platform applications, etc.

[0032] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting Internet access, including but not limited to desktop computers, smart phones, tablet computers, smart watches, notebook computers, laptop computers, e-book readers, etc.

[0033] The server 105 may be a server that provides various information query services, such as an e-commerce service platform server or a search engine server. The server 105 may receive query requests sent by the terminal devices 101, 102, 103, find relevant information according to the query conditions defined in the query request, and feed back the found information to the terminal devices 101, 102, 103 via the network 104.

[0034] The terminal devices 101, 102, 103 may include components for performing physical operations (eg, processors such as GPUs), and the terminal devices 101, 102, 103 may also perform local processing on the similar image search initiated by the user 110 to obtain similar image search results.

[0035] The method for determining similar pictures provided in the embodiment of the present application can be executed by the terminal device 101, 102, 103 or the server 105, and accordingly, the device for determining similar pictures can be set in the terminal device 101, 102, 103 or the server 105.

[0036] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the embodiment is only for illustration purposes. Any number of terminal devices, networks, and servers may be provided as required.

[0037] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a method for determining similar pictures according to the present application. The method for determining similar pictures comprises the following steps:

[0038] Step 201: Input the target image into the feature extraction network in the trained image similarity evaluation model to extract image features, and obtain a feature vector of the target image.

[0039] In this embodiment, the execution subject (eg Figure 1 The target image can be a picture selected by the user, for example, one or more pictures on a web page selected by the user when browsing the web page; the target image can also be a picture randomly extracted from a network picture library, for example, when it is desired to analyze the relevant information of an item provided by an e-commerce platform in a competitor's platform or store, a picture can be extracted from the description pictures of the item provided by the e-commerce platform as the target picture.

[0040] The target image may be input into a trained image similarity evaluation model. The image similarity evaluation model may be a model for evaluating the similarity between two or more images, and may be obtained by pre-training based on a sample set. The image similarity model may include a feature extraction network, which may extract features of each of the two or more input images, and the image similarity model may also calculate the similarity between the images based on the extracted features.

[0041] The feature extraction network may be a deep neural network, for example, a deep convolutional neural network. When training the image similarity evaluation model, the parameters of the feature extraction network may be iteratively adjusted so that the feature extraction network can extract features with stronger descriptive power for the image, and the features extracted by the feature extraction network can more effectively distinguish similar images from dissimilar images.

[0042] In this embodiment, after the target image is input into the image similarity evaluation model, the feature extraction network in the image similarity evaluation model can extract the features of the target image to obtain a feature vector of the target image.

[0043] Step 202: Based on the feature vector of the target picture and the feature matrix of the candidate picture set obtained in advance, determine similar pictures of the target picture from the candidate picture set.

[0044] Here, the feature matrix of the candidate picture set is constructed by the feature vectors of each candidate picture in the candidate picture set, and the feature vectors of the candidate pictures are obtained by extracting features from the candidate pictures based on the feature extraction network in the trained image similarity evaluation model.

[0045] The candidate images in the candidate image set can be images on the Internet or images in a pre-specified image library. In practice, when the target image is a description image of an item on an e-commerce platform, item description images on the same platform or other platforms can be collected to form a candidate image set. For example, a crawler program can be used to capture item data on each platform. Optionally, data cleaning can be performed on these item data to obtain relevant information of the candidate images, and then stored in a database after structured processing. For example, after the item data is cleaned, the image address, the item identifier described by the image, the item price, the update time of the item description information, the identifier of the platform where the image is located, the image identifier (such as the image ID), etc. can be extracted and stored in the database.

[0046] Furthermore, feature extraction can be performed on the candidates in the candidate image set in advance to obtain a feature vector of each candidate image. Specifically, the candidate image can be input into a trained image similarity evaluation model, and the output of the feature extraction network can be used as the feature vector of the candidate image. Then, a feature matrix of the candidate image set can be constructed based on the feature vectors of each candidate image. For example, the feature vectors of each candidate image can be combined into a feature matrix by row or column, and the row number or column number of the feature matrix can be associated with the corresponding candidate image.

[0047] In this embodiment, the similarity between the target image and the candidate image can be calculated using the feature vector of the target image and the feature matrix of the candidate image set, and similar images of the target image can be determined from the candidate image set according to the similarity calculation result. Among them, similarity calculation can be performed using similarity metrics such as cosine similarity, Pearson correlation coefficient, and Euclidean distance.

[0048] Specifically, if the feature matrix of the candidate picture set is formed by combining the feature vectors of each candidate picture in rows or columns, then the cosine similarity, Pearson correlation coefficient and other similarity measurement parameters between the feature vector of the target picture and each row vector or column vector in the feature matrix can be calculated as the similarity between the target picture and the corresponding candidate picture. The similarity between the feature vector of the target picture and the feature vector of the candidate picture represented by each row element or column element in the feature matrix is ​​calculated in sequence. Then, the calculated similarities with the feature vectors of each candidate picture can be sorted, and the candidate pictures with preset positions before sorting can be selected as similar pictures of the target picture; or the candidate pictures with similarities greater than a preset similarity threshold can be selected as similar pictures of the target picture.

[0049] In the above method, by constructing a feature matrix of a candidate image set, the features of a large number of candidate images can be orderly represented. In some optional implementations, after extracting a feature vector of a candidate image or a feature vector of a target image using a feature extraction network in a trained image similarity evaluation model, the extracted feature vector can be normalized to accurately obtain a similarity measure between the target image and the candidate image based on a unified similarity evaluation standard when determining similar images of the target image.

[0050] The method for determining similar images in the above-mentioned embodiment of the present application uses the feature extraction network in the trained image similarity evaluation model to extract features from the target image and the candidate image, and determines the candidate image similar to the target image based on the extracted feature vector. Since the trained image similarity evaluation model can estimate the similarity between images more accurately, the feature vector extracted by the feature extraction network has better descriptive power for the image features used as the basis for determining similar images, and thus can improve the accuracy of similar image search. At the same time, since there is no need to perform operations such as word segmentation and indexing on the description data of the image in advance, the dependence on the data index mode of the search engine is reduced, and similar images of the target image can be determined quickly and efficiently.

[0051] An exemplary application scenario of the above-mentioned embodiment of the present application is: when a user visits an e-commerce platform to browse a picture of an item, if the user hopes to find relevant information of other similar items on the e-commerce platform or hopes to view relevant information of the same item or similar items on other e-commerce platforms, the description picture of the currently browsed item can be used as the target picture, and a similar picture search request can be initiated to the background server. The background server can use the feature extraction network in the trained image similarity evaluation model to extract the feature vector of the target picture, and obtain a feature matrix constructed by the feature vectors of the item pictures collected on the e-commerce platform, and calculate the similarity between the target picture and the item pictures collected on the e-commerce platform according to the feature vector of the target picture and the feature matrix, so as to determine similar pictures and then find similar items to the currently browsed item.

[0052] In some optional implementations of the above embodiment, the feature vector of the candidate image may be a normalized row feature vector, and in the feature matrix of the candidate image set, the feature vector of each candidate image is a row vector of the feature matrix, and the feature vector of the target image may be a normalized column feature vector. For example, the feature matrix of the candidate image set is A n =(α1,α2,α3,…,α n ) T , the normalized column eigenvector of the target image is β, where n is the number of rows of the feature matrix, that is, the total number of candidate images corresponding to the eigenvectors of the feature matrix, αi is the normalized row feature vector of the i-th candidate image, i = 1, 2, 3, ..., n. Since the feature vectors of the candidate images and the target images are extracted by the feature extraction network in the trained image similarity evaluation model, the feature vectors of the candidate images and the target images have the same dimension, that is, the vector α i Same dimension as β.

[0053] In the above step 202, similar pictures of the target picture can also be determined in the following manner: determine the maximum value of the elements in the result column vector obtained by multiplying the feature matrix and the feature vector of the target picture, and determine that the candidate picture represented by the feature vector corresponding to the position of the maximum value in the result column vector in the feature matrix is ​​the similar picture of the target picture. Specifically, the feature matrix A can be n Multiply it with the eigenvector β to get the resulting column vector A n ×β=(α1β,α2β,α3β,…,α n β) T , then determine the maximum value of the elements in the resulting column vector and the row number where the maximum value is located, that is, in α1β, α2β, α3β, …, α n Determine the maximum value max(α i β) and the maximum value max(α i β) corresponds to the row number argmax(α i β), the row number in the candidate image feature matrix is ​​argmax(α i The candidate image represented by the feature vector of β) is determined as a similar image to the target image.

[0054] Similarly, in some other optional implementations of this embodiment, the feature vectors of the candidate images may be normalized column feature vectors, and in the feature matrix of the candidate image set, the feature vector of each candidate image is a column vector of the feature matrix, and the feature vector of the target image may be a normalized row feature vector. For example, the feature matrix of the candidate image set is B m =(β1,β2,β3,…,β m ), the normalized row feature vector of the target image is α, where m is the number of columns of the feature matrix, that is, the total number of candidate images corresponding to the feature vectors of the feature matrix; β j is the normalized column feature vector of the j-th candidate image, j = 1, 2, 3, …, m.

[0055] In the above step 202, similar pictures of the target picture can also be determined in the following manner: determine the maximum value of the elements in the resultant row vector obtained by multiplying the feature vector of the target picture with the feature matrix, and determine that the candidate picture represented by the feature vector corresponding to the position of the maximum value in the resultant row vector in the feature matrix is ​​the similar picture of the target picture. Specifically, the feature vector α and the feature matrix B can be m Multiply to get the result row vector α×B m =(αβ1,αβ2,αβ3,…,αβ m ), then determine the maximum value of the elements in the resulting row vector and the row number where the maximum value is located, that is, in αβ1, αβ2, αβ3, …, αβ m Determine the maximum value max(αβ j ) and the maximum value max(αβ j ) corresponds to the row number argmax(αβ j ), the row number in the above candidate image feature matrix is ​​argmax(αβ j ) is determined as a similar picture of the target picture.

[0056] By searching for the maximum value in the column vector or row vector obtained by multiplying the feature matrix of the candidate image set and the feature vector of the target image, and corresponding the position of the maximum value in the column vector or row vector obtained by multiplication to the feature matrix, the feature vector of the image similar to the target image is obtained, so that the similar images of the target image can be quickly determined by matrix multiplication operation, further improving the efficiency of similar image search.

[0057] Continue to refer Figure 3 , which shows a flow chart of another embodiment of the method for determining similar pictures according to the present application. Figure 3 As shown, the process 300 of the method for determining similar pictures in this embodiment includes the following steps:

[0058] Step 301 : derive a trained image similarity evaluation model based on a set of sample image pairs constructed from candidate images.

[0059] In this embodiment, the image similarity evaluation model includes a similarity calculation network and two feature extraction networks that share weights. The feature extraction network can be a neural network, for example, a multi-layer convolutional neural network, and the weights of the two feature extraction networks in the image similarity evaluation model can be shared. In addition, the two feature extraction networks have the same structure. During the training process, when adjusting the parameters of the image similarity evaluation model to be trained, the weights of the two feature extraction networks are adjusted synchronously.

[0060] A crawler program can be used to crawl pictures on the Internet to form a candidate picture set and construct a sample picture pair set of an image similarity evaluation model. Here, the sample picture pair in the sample picture pair set can be composed of two candidate pictures. Operators such as SIFT (Scale-invariant feature transform) can be used to extract features of the sample picture pair, and the similarity of the sample picture pair is calculated based on the extracted features as the similarity annotation information of the sample picture pair.

[0061] First, an image similarity evaluation model to be trained can be constructed based on the twin network, and then a sample image pair is input into the image similarity evaluation model to be trained to obtain the similarity evaluation results of the two candidate images in the sample image pair, and the similarity evaluation results are compared with the similarity annotation information of the sample image pair. According to the difference between the two, the parameters in the image similarity evaluation model to be trained are iteratively adjusted, including adjusting the shared weights of the two feature extraction networks. After that, the operation of inputting the sample image pair into the image similarity evaluation model to be trained, obtaining the similarity evaluation results of the two candidate images in the sample image pair, and then comparing them with the similarity annotation information of the sample image pair, and iteratively adjusting the parameters in the image similarity evaluation model to be trained according to the difference between the two is repeated. When the value of the loss function reaches the preset convergence condition after iteration, the parameters of the image similarity evaluation model to be trained are fixed to obtain a trained image similarity evaluation model.

[0062] Please refer to Figure 4 , which shows that according to Figure 3 The schematic diagram of the structure of the image similarity evaluation model in the method for determining similar pictures is shown in FIG. Figure 4 As shown, the image similarity evaluation model may include two feature extraction networks with shared weights and a similarity calculation network. The two feature extraction networks may extract features from the input images I1 and I2 respectively, and the extracted features are input into the similarity calculation network to obtain the similarity Sim(I1, I2) between the input images I1 and I2.

[0063] In some optional implementations of this embodiment, the above-mentioned sample picture pair set may include a sample picture pair formed by candidate pictures and similar attribute annotation information of the sample picture pair. Specifically, the candidate pictures in the candidate picture pair set may be combined in pairs to form a sample picture pair set, and the manually annotated label used to characterize whether the two pictures in the sample picture pair are similar is used as the similar attribute annotation information of the sample picture pair, for example, when the two pictures in the sample picture pair are similar, the label is "1", and when they are not similar, the label is "0".

[0064] At this time, based on the set of sample picture pairs, a trained image similarity evaluation model can be obtained by training in the following manner: two sample pictures in the sample picture pair are respectively input into two feature extraction networks in the image similarity evaluation model to be trained to obtain feature vectors of the two sample pictures in the sample picture pair; based on the feature vectors of the two sample pictures in the sample picture pair, the similarity of the two sample pictures in the sample picture pair is calculated using the similarity calculation network in the image similarity evaluation model to be trained; based on a preset loss function, the weights of the feature extraction network in the image similarity evaluation model to be trained are iteratively adjusted using a back propagation method so that the value of the loss function satisfies a preset convergence condition, wherein the value of the loss function is used to characterize the difference between the similarity of the sample picture pair calculated by the image similarity evaluation model to be trained and the annotation information of the similar attributes of the corresponding sample picture pair.

[0065] Specifically, in the training process of the image similarity evaluation model, first, an image similarity evaluation model to be trained including a similarity calculation network and two feature extraction networks with shared weights can be constructed, and then two sample images in the sample image pair are respectively input into the two feature extraction networks for feature extraction, and then based on the Euclidean distance between the two extracted feature vectors in the similarity calculation network. Specifically, the value of a preset loss function can be calculated, and the loss function can be constructed based on the Euclidean distance between the two extracted feature vectors. For example, the loss function can be constructed based on the similarity between the two sample images represented by the Euclidean distance and the similar attribute annotation information of the two sample images (i.e., the labels of whether the two sample images are similar).

[0066] Alternatively, the loss function L can be constructed according to formula (1):

[0067]

[0068] Wherein, d is the Euclidean distance between two sample images in a sample image pair obtained by the image similarity model to be trained, y is a label indicating whether the two sample images in the sample image pair are similar, y=1 indicates that the two sample images are similar, y=0 indicates that the two sample images are not similar, margin is the set threshold, and N is the number of sample image pairs.

[0069] According to formula (1), it can be seen that when the Euclidean distance between two sample images in the sample image pair calculated by the image similarity evaluation model to be trained is consistent with the similarity attribute between the two sample images indicated by the similarity attribute annotation information, the value of the loss function is smaller, and vice versa. The value of the loss function can effectively represent the accuracy of the similarity evaluation result of the image similarity evaluation model to be trained.

[0070] During the training process, a method such as gradient descent can be used to back-propagate the value of the loss function to the image similarity evaluation model to be trained, and the parameters of the image similarity evaluation model to be trained can be iteratively adjusted. The parameters of the image similarity evaluation model can include the weights of the feature extraction network. After adjusting the parameters, the image similarity evaluation model is reused to evaluate the similarity of the sample image pairs, and then the evaluation results of the image similarity evaluation model are continuously compared with the similar attribute annotation information using the loss function value, and the parameters are continuously adjusted according to the comparison results until the error of the similarity evaluation result of the image similarity evaluation model to be trained after the adjustment of the parameters converges, or the number of iterative adjustments reaches a preset number, and the trained image similarity evaluation model is obtained.

[0071] By constructing a set of sample picture pairs based on candidate pictures to train a trained image similarity evaluation model, a training sample with a larger number of samples can be constructed, and since the image similarity evaluation model is trained based on candidate pictures, the feature extraction network in the trained image similarity evaluation model can more accurately extract the features of the candidate pictures, and the extracted feature vectors can also better distinguish dissimilar pictures.

[0072] In addition, by obtaining the similar attribute annotation information of the sample image pairs when constructing the training samples, the image similarity evaluation model can use the similar attribute annotation information as the expected similarity evaluation result, quickly learn the logic of image feature extraction and similarity evaluation, and effectively improve the training speed.

[0073] In some optional implementations, after step 301, user feedback information on the similarity evaluation between the target image and the similar image of the target image can also be obtained; similar attribute annotation information between the target image and the similar image of the target image is determined based on the user feedback information, and the target image and the similar image of the target image are added as a sample image pair to the sample image pair set.

[0074] Specifically, the determined similar pictures can be pushed to the user who issued the request to find similar pictures of the target picture, and then the user's feedback information on whether the pushed similar pictures are similar to the target picture is collected. Then the target picture and the pushed similar pictures are used as sample picture pairs, and the user's feedback information is used as the similarity attribute annotation information of this sample picture pair, and added to the sample picture pair set. In this way, the sample picture pair set can be continuously updated using user feedback information, and then the image similarity evaluation model can be iteratively updated to improve the quality of the image similarity evaluation model.

[0075] return Figure 3 In step 302, the target image is input into the feature extraction network in the trained image similarity evaluation model to extract image features and obtain a feature vector of the target image.

[0076] The target image can be input into the image similarity evaluation model trained in step 301, and the feature vector of the target image can be extracted using the feature extraction network in the trained image similarity evaluation model.

[0077] In step 303, based on the feature vector of the target picture and the feature matrix of the candidate picture set obtained in advance, similar pictures to the target picture are determined from the candidate picture set.

[0078] The feature matrix of the candidate picture set is constructed by the feature vectors of each candidate picture in the candidate picture set, and the feature vectors of the candidate pictures are obtained by extracting features from the candidate pictures based on the feature extraction network in the trained image similarity evaluation model.

[0079] The feature vectors of each candidate image output by the feature extraction network at the last iteration during the training of the image similarity evaluation model can be obtained, and a feature matrix of the candidate image set can be constructed. Alternatively, the candidate images in the candidate image set can be re-input into the feature extraction network in the trained image similarity evaluation model to obtain the feature vectors of the candidate images, and then the feature matrix of the candidate image set can be constructed.

[0080] Optionally, the feature vectors of each candidate image in the candidate image set and the feature vector of the target image can be normalized, so that when calculating the similarity between the target image and each candidate image, the similarity between the target image and different candidate images is determined based on the normalized similarity evaluation standard. For example, when using the Euclidean distance to select the candidate image with the highest similarity to the target image, the normalized feature vector can more accurately describe the similarity between the two.

[0081] Step 302 and step 303 of this embodiment are respectively consistent with step 201 and step 202 of the aforementioned embodiment. The specific implementation of step 302 and step 303 can also refer to the description of the implementation of step 201 and step 202 in the aforementioned embodiment, and will not be repeated here.

[0082] Figure 3 The method for determining similar images shown can further improve the accuracy of similar image search by adding a step of using a set of candidate image pairs to train an image similarity evaluation model including a similarity calculation network and two feature extraction networks with shared weights.

[0083] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for determining similar pictures. Figure 2 and Figure 3Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0084] like Figure 5 As shown, the apparatus 500 for determining similar pictures of this embodiment includes an extraction unit 501 and a determination unit 502. The extraction unit 501 can be configured to input the target picture into the feature extraction network in the trained image similarity evaluation model to extract image features and obtain a feature vector of the target picture; the determination unit 502 can be configured to determine similar pictures of the target picture from the candidate picture set based on the feature vector of the target picture and the feature matrix of the candidate picture set obtained in advance. The feature matrix of the candidate picture set is constructed by the feature vectors of each candidate picture in the candidate picture set, and the feature vectors of the candidate pictures are obtained by extracting features of the candidate pictures based on the feature extraction network in the trained image similarity evaluation model.

[0085] In some embodiments, the feature vector of the candidate image is a normalized row feature vector, and the feature vector of the target image is a normalized column feature vector; in the feature matrix of the candidate image set, the feature vector of each candidate image is a row vector in the feature matrix; the determination unit 502 can be further configured to determine similar images of the target image from the candidate image set based on the feature vector of the target image and the feature matrix of the candidate image set obtained in advance in the following manner: determine the maximum value of the elements in the result column vector obtained by multiplying the feature matrix by the feature vector of the target image, and determine that the candidate image represented by the feature vector corresponding to the position of the maximum value in the result column vector in the feature matrix is ​​a similar image of the target image.

[0086] In some embodiments, the feature vector of the candidate image is a normalized column feature vector, and the feature vector of the target image is a normalized row feature vector; in the feature matrix of the candidate image set, the feature vector of each candidate image is a column vector in the feature matrix; the determination unit 502 can be further configured to determine similar images of the target image from the candidate image set based on the feature vector of the target image and the feature matrix of the candidate image set obtained in advance in the following manner: determine the maximum value of the elements in the resultant row vector obtained by multiplying the feature vector of the target image by the feature matrix, and determine that the candidate image represented by the feature vector corresponding to the position of the maximum value in the resultant row vector in the feature matrix is ​​a similar image of the target image.

[0087] In some embodiments, the image similarity evaluation model includes a similarity calculation network and two feature extraction networks with shared weights. The apparatus 500 may also include a training unit configured to obtain a trained image similarity evaluation model based on a set of sample image pairs constructed from candidate images.

[0088] In some embodiments, the above-mentioned sample picture pair set includes a sample picture pair formed by candidate pictures and similar attribute annotation information of the sample picture pair; the above-mentioned training unit can be further configured to train and obtain a trained image similarity evaluation model in the following manner: input two sample pictures in the sample picture pair into two feature extraction networks in the image similarity evaluation model to be trained respectively, and obtain feature vectors of the two sample pictures in the sample picture pair; based on the feature vectors of the two sample pictures in the sample picture pair, use the similarity calculation network in the image similarity evaluation model to be trained to calculate the similarity of the two sample pictures in the sample picture pair; based on a preset loss function, use the back propagation method to iteratively adjust the weights of the feature extraction network in the image similarity evaluation model to be trained, so that the value of the loss function satisfies the preset convergence condition, wherein the value of the loss function is used to characterize the difference between the similarity of the sample picture pair calculated by the image similarity evaluation model to be trained and the annotation information of the similar attributes of the corresponding sample picture pair.

[0089] In some embodiments, the device 500 may also include an updating unit configured to: obtain user feedback information on the similarity evaluation between the target image and the similar image of the target image; determine the similar attribute annotation information between the target image and the similar image of the target image based on the user feedback information, and add the target image and the similar image of the target image as a sample image pair to the sample image pair set.

[0090] It should be understood that the units described in the apparatus 500 are similar to those described in the reference Figure 2 and Figure 3 Therefore, the operations and features described above for the method are also applicable to the device 500 and the units included therein, and will not be described in detail here.

[0091] The device 500 for determining similar images in the above-mentioned embodiment of the present application performs feature extraction on the target image and the candidate image by utilizing the feature extraction network in the trained image similarity evaluation model, and determines the candidate image similar to the target image based on the extracted feature vector, thereby improving the efficiency and accuracy of similar image search.

[0092] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 suitable for implementing an electronic device of an embodiment of the present application. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0093] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0094] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0095] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0096] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor includes an extraction unit and a determination unit. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances. For example, the extraction unit may also be described as "a unit that inputs the target image into the feature extraction network in the trained image similarity evaluation model to extract image features and obtain a feature vector of the target image".

[0099] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device inputs the target image into the feature extraction network in the trained image similarity evaluation model to extract image features and obtain a feature vector of the target image; based on the feature vector of the target image and the feature matrix of the candidate image set obtained in advance, a similar image of the target image is determined from the candidate image set; wherein the feature matrix of the candidate image set is constructed by the feature vectors of each candidate image in the candidate image set, and the feature vectors of the candidate images are obtained by extracting features of the candidate images based on the feature extraction network in the trained image similarity evaluation model.

[0100] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for determining similar pictures, comprising: Input the target image into the feature extraction network in the trained image similarity evaluation model to extract image features, and obtain a feature vector of the target image; wherein the target image is a description image of the item currently browsed by the user provided by the e-commerce platform; Based on the feature vector of the target image and the feature matrix of the candidate image set obtained in advance, determine similar images of the target image from the candidate image set; wherein the candidate image set is formed by item description images collected from other e-commerce platforms different from the e-commerce platform, and the similar images are used to find similar items to the currently browsed item; Among them, the trained image similarity evaluation model is obtained by training based on candidate images in the candidate image set; the feature matrix of the candidate image set is constructed based on the feature vectors of the candidate images output by the feature extraction network of the image similarity evaluation model at the last iteration during the training process of the image similarity evaluation model.

2. The method according to claim 1, wherein: The feature vector of the candidate image is a normalized row feature vector, and the feature vector of the target image is a normalized column feature vector; In the feature matrix of the candidate picture set, the feature vector of each candidate picture is a row vector in the feature matrix; The determining, based on the feature vector of the target picture and the feature matrix of the candidate picture set obtained in advance, a similar picture of the target picture from the candidate picture set comprises: Determine the maximum value of the elements in the result column vector obtained by multiplying the feature matrix and the feature vector of the target image, and determine that the candidate image represented by the feature vector in the feature matrix corresponding to the position of the maximum value in the result column vector is a similar image of the target image.

3. The method according to claim 1, wherein: The feature vector of the candidate image is a normalized column feature vector, and the feature vector of the target image is a normalized row feature vector; In the feature matrix of the candidate picture set, the feature vector of each candidate picture is a column vector in the feature matrix; The determining, based on the feature vector of the target picture and the feature matrix of the candidate picture set obtained in advance, a similar picture of the target picture from the candidate picture set comprises: Determine the maximum value of the elements in the resultant row vector obtained by multiplying the feature vector of the target image by the feature matrix, and determine that the candidate image represented by the feature vector corresponding to the position of the maximum value in the resultant row vector in the feature matrix is ​​a similar image to the target image.

4. The method according to any one of claims 1 to 3, wherein: The image similarity evaluation model includes a similarity calculation network and two feature extraction networks with shared weights; The method further comprises: The trained image similarity evaluation model is obtained based on training of a set of sample image pairs constructed from candidate images.

5. The method according to claim 4, wherein: The sample picture pair set includes sample picture pairs formed by candidate pictures and similar attribute annotation information of the sample picture pairs; The trained image similarity evaluation model is obtained by training a set of sample pictures constructed based on candidate pictures, including: Inputting two sample images in the sample image pair into two feature extraction networks in the image similarity evaluation model to be trained respectively, and obtaining feature vectors of the two sample images in the sample image pair; Based on the feature vectors of the two sample images in the sample image pair, the similarity of the two sample images in the sample image pair is calculated using a similarity calculation network in the image similarity evaluation model to be trained; Based on a preset loss function, a back-propagation method is used to iteratively adjust the weights of the feature extraction network in the image similarity evaluation model to be trained so that the value of the loss function satisfies a preset convergence condition, wherein the value of the loss function is used to characterize the difference between the similarity of the sample image pairs calculated by the image similarity evaluation model to be trained and the annotation information of the similar attributes of the corresponding sample image pairs.

6. The method according to claim 5, wherein: The method further comprises: Obtaining user feedback information on the similarity evaluation between the target image and the similar images of the target image; Based on the user feedback information, similar attribute labeling information between the target image and similar images of the target image is determined, and the target image and the similar images of the target image are added as sample image pairs to the sample image pair set.

7. A device for determining similar pictures, comprising: An extraction unit is configured to input a target image into a feature extraction network in a trained image similarity evaluation model to extract image features, thereby obtaining a feature vector of the target image; wherein the target image is a description image of an item currently browsed by a user provided by an e-commerce platform; A determination unit is configured to determine similar pictures of the target picture from the candidate picture set based on the feature vector of the target picture and a feature matrix of the candidate picture set obtained in advance; wherein the candidate picture set is formed by item description pictures collected from other e-commerce platforms different from the e-commerce platform, and the similar pictures are used to find similar items to the currently browsed item; Among them, the trained image similarity evaluation model is obtained by training based on candidate images in the candidate image set; the feature matrix of the candidate image set is constructed based on the feature vectors of the candidate images output by the feature extraction network of the image similarity evaluation model at the last iteration during the training process of the image similarity evaluation model.

8. The device according to claim 7, wherein: The feature vector of the candidate image is a normalized row feature vector, and the feature vector of the target image is a normalized column feature vector; In the feature matrix of the candidate picture set, the feature vector of each candidate picture is a row vector in the feature matrix; The determining unit is further configured to determine similar pictures of the target picture from the candidate picture set based on the feature vector of the target picture and the feature matrix of the candidate picture set obtained in advance in the following manner: Determine the maximum value of the elements in the result column vector obtained by multiplying the feature matrix and the feature vector of the target image, and determine that the candidate image represented by the feature vector in the feature matrix corresponding to the position of the maximum value in the result column vector is a similar image of the target image.

9. The device according to claim 7, wherein: The feature vector of the candidate image is a normalized column feature vector, and the feature vector of the target image is a normalized row feature vector; In the feature matrix of the candidate picture set, the feature vector of each candidate picture is a column vector in the feature matrix; The determining unit is further configured to determine similar pictures of the target picture from the candidate picture set based on the feature vector of the target picture and the feature matrix of the candidate picture set obtained in advance in the following manner: Determine the maximum value of the elements in the resultant row vector obtained by multiplying the feature vector of the target image by the feature matrix, and determine that the candidate image represented by the feature vector corresponding to the position of the maximum value in the resultant row vector in the feature matrix is ​​a similar image to the target image.

10. The device according to any one of claims 7 to 9, wherein: The image similarity evaluation model includes a similarity calculation network and two feature extraction networks with shared weights; The device also includes: The training unit is configured to obtain the trained image similarity evaluation model based on the sample image pair set constructed by the candidate images.

11. The device according to claim 10, wherein: The sample picture pair set includes sample picture pairs formed by candidate pictures and similar attribute annotation information of the sample picture pairs; The training unit is further configured to train the trained image similarity evaluation model in the following manner: Inputting two sample images in the sample image pair into two feature extraction networks in the image similarity evaluation model to be trained respectively, and obtaining feature vectors of the two sample images in the sample image pair; Based on the feature vectors of the two sample images in the sample image pair, the similarity of the two sample images in the sample image pair is calculated using a similarity calculation network in the image similarity evaluation model to be trained; Based on a preset loss function, a back-propagation method is used to iteratively adjust the weights of the feature extraction network in the image similarity evaluation model to be trained so that the value of the loss function satisfies a preset convergence condition, wherein the value of the loss function is used to characterize the difference between the similarity of the sample image pairs calculated by the image similarity evaluation model to be trained and the annotation information of the similar attributes of the corresponding sample image pairs.

12. The device according to claim 11, wherein: The apparatus further comprises an updating unit, wherein the updating unit is configured to: Obtaining user feedback information on the similarity evaluation between the target image and the similar image of the target image; Based on the user feedback information, similar attribute labeling information between the target image and similar images of the target image is determined, and the target image and the similar images of the target image are added as sample image pairs to the sample image pair set.

13. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

14. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images