Image processing method, device and storage medium

Abstract

The embodiment of the present invention discloses an image processing method, device and storage medium. The method includes: extracting corresponding semantic features from a sample image through a first neural network model; clustering the semantic features extracted from the sample image to obtain a semantic label corresponding to the sample image; training a second neural network model through the sample image and the corresponding semantic label; extracting corresponding semantic features from a first image in a first gallery and a second image in a second gallery through the second neural network model; determining a target semantic distance for distinguishing the first image from the second gallery based on the semantic features extracted from the first image and the second image; retrieving a first image that is identical to the received image to be retrieved from the first gallery based on the target semantic distance. Through the embodiment of the present invention, the accuracy and efficiency of image retrieval can be improved.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] Deep learning technology based on neural network models has developed rapidly, especially in the field of image retrieval. Many image retrieval task scenarios have introduced retrieval algorithms based on deep learning. When a user inputs an image to be retrieved, the neural network model can recognize the input image and retrieve a large number of images as retrieval results based on a fixed matching threshold set by the user. Among them, the retrieval results are suspected similar images, and the user needs to make a manual secondary judgment to find the same image as the image to be retrieved from the retrieval results. It is impossible to accurately retrieve the same image as the image to be retrieved. Summary of the invention

[0003] The embodiments of the present invention provide an image processing method, an apparatus and a storage medium, which can improve the accuracy and efficiency of image retrieval.

[0004] The technical solution of the embodiment of the present invention is achieved as follows:

[0005] An embodiment of the present invention provides an image processing method, including:

[0006] Extracting corresponding semantic features from the sample image through the first neural network model;

[0007] Clustering the semantic features extracted from the sample images to obtain semantic labels corresponding to the sample images;

[0008] Training a second neural network model using the sample images and corresponding semantic labels;

[0009] Extracting corresponding semantic features from the first image in the first gallery and from the second image in the second gallery by the second neural network model;

[0010] Determining a target semantic distance for distinguishing the first image from the second image gallery according to semantic features extracted from the first image and the second image;

[0011] A first image that is the same as the received image to be retrieved is retrieved from the first gallery according to the target semantic distance.

[0012] An embodiment of the present invention provides an image processing method, including:

[0013] comparing a semantic distance between a first image in a first gallery and a second image in a second gallery to determine a target semantic distance for distinguishing the first image from the second gallery;

[0014] Retrieving a first image that is the same as the received image to be retrieved from the first gallery according to the target semantic distance;

[0015] Output the search result corresponding to the image to be searched.

[0016] An embodiment of the present invention provides an image processing device, including:

[0017] A first extraction unit, used to extract corresponding semantic features from the sample image through a first neural network model;

[0018] A clustering unit, used for clustering the semantic features extracted from the sample image to obtain a semantic label corresponding to the sample image;

[0019] A training unit, used to train a second neural network model using the sample images and corresponding semantic labels

[0020] A second extraction unit, configured to extract corresponding semantic features from the first image in the first gallery and from the second image in the second gallery by using the second neural network model;

[0021] a distinguishing unit, configured to determine a target semantic distance for distinguishing the first image from the second image gallery based on semantic features extracted from the first image and the second image;

[0022] A retrieval unit is used to retrieve a first image that is the same as the received image to be retrieved from the first image library according to the target semantic distance.

[0023] In some embodiments, the retrieval unit is further used to:

[0024] Determining a semantic distance between the received image to be retrieved and the first image in the first gallery;

[0025] When the semantic distance is less than the target semantic distance of the first image, the corresponding first image is identified as a retrieval result corresponding to the image to be retrieved.

[0026] In some embodiments, the training unit is further used to:

[0027] According to the sample image and the corresponding semantic label, the parameters of the second neural network model are updated until the loss function of the second neural network model converges; wherein the complexity of the first neural network model is greater than the complexity of the second neural network model.

[0028] An embodiment of the present invention provides an image processing device, including:

[0029] a determining unit, configured to compare a semantic distance between a first image in a first gallery and a second image in a second gallery to determine a target semantic distance for distinguishing the first image from the second gallery;

[0030] A matching unit, configured to retrieve, from the first image library, a first image identical to the received image to be retrieved according to the target semantic distance;

[0031] The output unit is used to output the search result corresponding to the image to be searched.

[0032] An embodiment of the present invention provides an image processing device, the device comprising: a memory and a processor, wherein:

[0033] The memory is used to store computer programs;

[0034] The processor is used to implement the steps of the above-mentioned image processing method when executing the computer program stored in the memory.

[0035] An embodiment of the present invention provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0036] The embodiments of the present invention have the following beneficial effects:

[0037] Based on the semantic features of each first image in the first gallery and the semantic features of each second image in the second gallery, information capable of distinguishing the first image from the second gallery, namely, the target semantic distance, is obtained. Thus, based on the target semantic distance as a standard, the first image having the same semantic features as the image to be retrieved can be searched from the first gallery as a retrieval result, thereby achieving good accuracy and efficiency of image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is an optional structural diagram of an image processing system according to an embodiment of the present invention;

[0039] Figure 2 is an optional structural diagram of the image retrieval system according to an embodiment of the present invention;

[0040] Figure 3 is an optional flowchart of the image processing method according to an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of semantic feature distribution in a feature space of an embodiment of the present invention;

[0042] Figure 5 is an optional flowchart of the image processing method according to an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the effect of pixel feature retrieval based on an embodiment of the present invention;

[0044] Figure 7 is a schematic diagram of semantic features of an embodiment of the present invention;

[0045] Figure 8 is a schematic diagram of the search effect of search scenario 2 according to an embodiment of the present invention;

[0046] Fig. 9 is an optional structural diagram of an image retrieval system in retrieval scenario 3 of an embodiment of the present invention;

[0047] Fig.10 It is a schematic diagram of an optional framework composition of the image processing method according to an embodiment of the present invention;

[0048] Fig.11 is an optional structural diagram of an image processing device according to an embodiment of the present invention;

[0049] Fig.12 It is an optional structural diagram of the image processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0051] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meanings as those commonly understood by those skilled in the art of the technical field of the embodiments of the present invention. The terms used in the embodiments of the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0053] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0054] 1) Neural network model, including a first neural network model and a second neural network model, wherein the second neural network model has a simpler structure than the first neural network model, characterized by fewer model parameters, fewer network layers, simple topological structure, and short paths. The first neural network model, i.e., a complex neural network model, has strong expressive power and robustness of image features, and may be a neural network model such as ResNet152, DenseNet164, and GoogleNet. The second neural network model, i.e., a lightweight neural network model, has the characteristics of fast computing speed, low computing overhead, and less resource usage, and may be a neural network model such as MobileNet and ShuffleNet.

[0055] 2) The first image library, also called the seed image library, is the image library based on which the image retrieval is based, and includes a plurality of first images (also called seed images). When performing image retrieval, it is necessary to retrieve first images similar to or identical to the target image in the first image library.

[0056] 3) The second image library, also called the white library, includes multiple second images, and the semantic content of the second images is different from that of the first images, or is unrelated (the semantic distance is greater than the semantic distance threshold). For example: the first image is an image of a car, and the second image is an image of a road, a tree, a street lamp, etc. For another example: the first image is an image of a trademark, and the second image is a landscape image.

[0057] 4) Image features: The image representation obtained by extracting features from an image through a neural network model can be either semantic features or pixel features. Semantic features represent the semantic content of an image, while pixel features represent the pixel information of an image.

[0058] 5) Clustering: A grouping obtained through cluster analysis, that is, a collection of objects with similar attributes. The attributes of objects in different clusters are not similar.

[0059] 6) Pseudo-labels: semantic labels obtained by clustering the image features of unlabeled training images can represent the classification of sample images.

[0060] The embodiments of the present invention may be provided as an image processing method, device and storage medium. In practical applications, the image processing method may be implemented by an image processing device, and each functional entity in the image processing device may be collaboratively implemented by hardware resources of a device (such as a terminal device, a server or a server cluster), such as computing resources such as a processor, and communication resources (such as for supporting various communication methods such as optical cables and cellular communications).

[0061] See also Figure 1 , Figure 11 is a schematic diagram of an exemplary architecture of an image processing system 100 provided in an embodiment of the present invention, wherein the image processing system 100 includes a first image library 11 and a second image library 12, a first neural network model 13 and a second neural network model 14. The image processing system 100 can be deployed on a server or a terminal.

[0062] The image processing system 100 inputs the sample image into the first neural network model 13, and the first neural network model 13 extracts the semantic features of the input sample image. The image processing system 100 clusters the semantic features of the sample image extracted by the first neural network model 13 to obtain the semantic labels corresponding to each sample image, and trains the second network model 14 based on the sample image and the corresponding semantic labels. In the trained second network model 14, the semantic features of the first image in the first gallery 11 and the semantic features of the second image in the second gallery 12 are respectively extracted. The image processing system 100 determines the target semantic distance for distinguishing the first image from the second gallery 12 as a whole based on the semantic features extracted by the second neural network model 14, that is, the target semantic features corresponding to each first image.

[0063] Exemplarily, the target semantic distance corresponding to the first image in the first gallery can be stored in the form of a list, as shown in List 16, the first image in the first gallery includes image 1, image 2, image 3 and image 4, and the target semantic distances of image 1, image 2, image 3 and image 4 are respectively: threshold 1, threshold 2, threshold 3, threshold 4. In practical applications, the storage form of the target semantic distance corresponding to the first image is not limited in any way.

[0064] It should be noted that Figure 1 The first gallery 11, the second gallery 12, the first neural network model 13 and the second neural network model 14 may be located on the same functional entity or on different functional entities.

[0065] When the image processing system 100 receives an image to be retrieved input by a user, a first image identical to the image to be retrieved is retrieved from the first image library 11 as a retrieval result according to the target semantic distance corresponding to the first image.

[0066] Here, the semantic distance between the image to be retrieved and each first image in the first gallery 11 can be determined, and the first image whose corresponding target semantic distance is greater than the semantic distance is used as the retrieval result of the image to be retrieved. For example, the semantic distance between the image to be retrieved and image 1 is d1, d1>threshold 1; the semantic distance between the image to be retrieved and image 2 is d2, d2>threshold 2; the semantic distance between the image to be retrieved and image 3 is d3, d3<threshold 3; the semantic distance between the image to be retrieved and image 4 is d4, d4>threshold 4; then image 3 is considered to be the same first image as the image to be retrieved, and the retrieval result of the image to be retrieved is image 3.

[0067] Exemplarily, an embodiment of the present invention further provides a data retrieval system, including: a terminal and a server, wherein an image retrieval system, a first image library and a second image library are running on the server or the terminal. The first image library and the second image library can be respectively Figure 1 The first image library 11 and the second image library 12 in the image processing system 100 are shown.

[0068] The image retrieval system compares the semantic distances between each first image in the first gallery and each second image in the second gallery to determine a target semantic distance for distinguishing the first image from the second gallery as a whole; retrieves a first image that is identical to the received image to be retrieved from the first gallery according to the target semantic distance; and outputs a retrieval result corresponding to the image to be retrieved.

[0069] In practical applications, the data retrieval system may further include a second neural network model, and the second neural network model in the data retrieval system may be the second neural network model 14 in the image processing system 100. Here, the image retrieval system and the image processing system may be integrated into the same application.

[0070] In an exemplary scenario, when the image retrieval system, the first gallery and the second gallery are deployed on a terminal, when the terminal receives an image to be retrieved input by a user, the retrieval result of the image to be retrieved is determined based on the target semantic distance corresponding to each first image in the first gallery.

[0071] In an exemplary scenario, when the image retrieval system, the first image library and the second image library are deployed on the server, such as Figure 2As shown, the data retrieval system includes a terminal 21, a server 22 and a network 23, and an image retrieval system 221, a first gallery 222 and a second gallery 223 are running in the server 22. When the terminal 21 receives the image to be retrieved input by the user, the image to be retrieved is sent to the server 22 through the network 23, and the image retrieval system 221 in the server 22 compares the semantic distance between each first image in the first gallery 222 and each second image in the second gallery 223 to determine the target semantic distance for distinguishing the first image from the second gallery 223; retrieves the first image that is the same as the received image to be retrieved from the first gallery 222 according to the target semantic distance; outputs the retrieved first image as a retrieval result to the terminal 21 through the network 23, and the terminal 21 displays the retrieval result to the user.

[0072] In practical applications, Figure 2 The server 22 shown may also include a second neural network 224, which may be Figure 1 A second neural network 14 is shown.

[0073] Among them, a mobile application (App) for image retrieval or a browsing web page can be installed on the terminal 21 as a client of the image retrieval system 221. The user can submit the image to be retrieved through the client, and the client sends the image to be retrieved to the server 22 through the network and receives the retrieval results returned by the server 22.

[0074] exist Figure 2 Only one server in the server 10 is shown as an example. Of course, it is not ruled out that multiple servers can be implemented to form a server cluster as the server 22.

[0075] It should be noted that the image processing method provided by the embodiment of the present invention is applicable to any scenario where similar or identical images need to be retrieved based on images. For example, in cloud album management, all images including user A are searched based on the image of a given user A; in a shopping website, all images of products of the same category as product A are searched based on the image of a given product A input by the user; for another example: in a trademark query system, trademarks similar or identical to product A are searched based on the trademark A input by the user.

[0076] Next, combine Figure 1 The image processing system and Figure 2 The data retrieval system shown is used to illustrate various embodiments of the image processing method, device, and computer-readable storage medium provided by the embodiments of the present invention.

[0077] See also Figure 3 , Figure 3 is a flowchart of an image processing method provided by an embodiment of the present invention, which will be combined with Figure 3 The steps shown are explained.

[0078] Step S301, extracting corresponding semantic features from the sample image through a first neural network model;

[0079] The image processing system inputs a sample image in the sample image set into a first neural network model, and extracts semantic features of each sample image in the first neural network model. The sample image may include a first image in a first gallery and a second image in a second gallery, and the sample image may be an image without a semantic label.

[0080] Here, the first neural network model may be a neural network model without a classifier, and the output result is a feature vector extracted from the input image, and the feature vector extracted from the input image is used as the semantic feature of the input image.

[0081] Step S302: clustering the semantic features extracted from the sample image to obtain a semantic label corresponding to the sample image.

[0082] The semantic features extracted from each sample image are clustered by a clustering algorithm, and the same semantic features are divided into closely related sets, so that the semantic features of the extracted sample images are divided into multiple sets, i.e., clusters, and the semantic labels of each set are determined, and the semantic labels of each set are used as the semantic labels of the sample images corresponding to the semantic features in the set. Here, the clustering algorithms used can be: K-means, mean shift, density-based spatial clustering of application with noise (DBSCAN), and other algorithms.

[0083] In the embodiment of the present invention, the semantic features of each sample image are clustered, and the semantic label of each sample image is determined according to the clustering result, without the need to annotate the sample images with semantic labels by manual annotation.

[0084] In some embodiments, clustering the semantic features extracted from the sample images to obtain semantic labels corresponding to the sample images includes: mapping the semantic features extracted from the sample images to a feature space; dividing the semantic features of the sample images into sets corresponding to different cluster centers based on the distances between the semantic features of the sample images and different cluster centers in the feature space; and uniformly assigning corresponding semantic labels to the sample images corresponding to the semantic features in the sets of the same cluster centers.

[0085] Map the semantic features of each sample image to the feature space, randomly select K points as the cluster centers of the K sets, and iteratively perform the following two steps until the cluster centers no longer change:

[0086] Step 1: Classify the collection

[0087] Semantic features are assigned to corresponding sets according to the distances between all semantic features and the cluster centers.

[0088] Step 2: Move the cluster center

[0089] The average value of the positions of the semantic features in each set is calculated, and the corresponding cluster center is moved to the position corresponding to the average value.

[0090] Through the above iterations, the semantic features of the sample images are divided into K sets, and the label of any sample image in each set is used as the semantic label of the corresponding set, and the semantic label is assigned to the sample image corresponding to the semantic feature in the set. Where K is greater than 2.

[0091] Step S303: training a second neural network model using the sample images and corresponding semantic labels.

[0092] After determining the semantic labels corresponding to the sample images in step S302, the second neural network model is trained using the sample images and the semantic labels corresponding to the sample images to update the parameters of the second neural network model.

[0093] In some embodiments, the complexity of the first neural network model is greater than the complexity of the second neural network model. The first neural network model may be a neural network model such as ResNet152, DenseNet164, GoogleNet, etc., and the second neural network model may be a neural network model such as MobileNet, ShuffleNet, etc.

[0094] In some embodiments, when training the second network model, the method further includes: updating the parameters of the second neural network model according to the sample image and the corresponding semantic label until the loss function of the second neural network model converges.

[0095] The parameters of the second neural network model are updated through the sample images and the semantic labels corresponding to each sample image, and the loss of the second neural network model after the parameter update is calculated based on the loss function. When the current loss decreases compared to the loss of the second neural network model before the update, the parameters of the second neural network model continue to be updated through the sample images and the semantic labels corresponding to each sample image until the loss value of the second neural network model no longer changes, that is, the second neural network model converges, and the training of the second neural network model is completed.

[0096] Here, the loss function used to train the second neural network model may be a mean square error function, a Huber loss function, a softmax loss function, a center loss function, or the like.

[0097] Step S304: extract corresponding semantic features from the first image in the first gallery and from the second image in the second gallery through the second neural network model.

[0098] After completing the training of the second neural network model, the semantic features of each first image in the first gallery and the semantic features of each second image in the second gallery are extracted in the second neural network model. The second neural network model may include multiple convolutional layers, fully connected layers and classifiers. The image processing system calls the second neural network model to extract feature vectors from each first image and each second image, and uses the feature vectors output by the convolutional layer or the fully connected layer as the semantic features of each first image and the semantic features of each second image.

[0099] The first images in the first gallery may be images with the same semantic content in the same scene, such as different face portraits in an authentication system, or images of different vehicles in a traffic system.

[0100] In some embodiments, the semantic features of the first image in the first gallery are not related to the semantic features of the second image in the second gallery. For example, the first image is a human face avatar, and the second image is an image of a tree, a chat interface, a car, etc., which has completely different semantics from a human face. Another example is that the first image is a human face avatar, and the second image is an image of a tree, a chat interface, a car, etc., which has a different pixel distribution from a human face.

[0101] Among them, the semantic content of the first image and the second image is different, and the pixel content may be the same. For example, the first image is a red flag image, and the second image is a red hat image. The color of the pixels of both is red, but the semantic content of the first image is the flag, and the semantic content of the second image is the hat.

[0102] Step S305: determining a target semantic distance for distinguishing the first image from the second image gallery according to the semantic features extracted from the first image and the second image.

[0103] According to the semantic features of each first image in the first gallery and the semantic features of each second image in the second gallery extracted in step S304, the target semantic distance corresponding to each first image is determined. The semantic distance represents the similarity between the semantic contents of the two corresponding images, and the target semantic distance represents the matching similarity threshold that distinguishes the corresponding first image and the second gallery as a whole, and represents the boundary of the semantic distance range between the first image and the second gallery.

[0104] In the embodiment of the present invention, the target semantic distance corresponding to each first image may be determined by at least one of the following two methods:

[0105] Method 1:

[0106] For each first image, the following processing is performed respectively to determine the target semantic distance corresponding to each first image: the semantic distance between the semantic features of the first image and the semantic features of different second images in the second image is calculated, the semantic distances between the first image and the different second images are compared, the semantic distances between the first image and the different second images are sorted, and the semantic distances of the set order are determined according to the sorting results as the target semantic distance corresponding to the first image.

[0107] The semantic distance between the semantic features of two images may be a Euclidean distance, a cosine distance, etc. between the two semantic features. The embodiment of the present invention does not impose any limitation on the calculation method of the semantic distance.

[0108] Method 2:

[0109] The semantic features of each first image and the semantic features of each second image extracted by the second neural network model are mapped to the feature space; and the target semantic distance corresponding to each first image is determined according to the distribution of the semantic features of each first image and the semantic features of each second image in the feature space.

[0110] Step S306: Retrieve a first image that is the same as the received image to be retrieved from the first image library according to the target semantic distance.

[0111] When an image to be retrieved is received, the semantic features of the image to be retrieved are extracted, and the semantic distance between the semantic features of the image to be retrieved and the semantic features of different first images in the first gallery is calculated, and the semantic distance between the image to be retrieved and the different first images is compared with the target semantic distance of the corresponding first image, and the first image as the retrieval result corresponding to the image to be retrieved is determined according to the comparison result. The semantic features of the image to be retrieved can be extracted by the second neural network model.

[0112] It should be noted that the embodiment of the present invention does not impose any limitation on the timing of receiving the image to be searched, for example, it may be before S301, before S302, or after S305, etc.

[0113] In some embodiments, retrieving a first image that is identical to the received image to be retrieved from the first gallery according to the target semantic distance includes: determining a semantic distance between the received image to be retrieved and the first image in the first gallery; when the semantic distance is less than the target semantic distance of the first image, identifying the corresponding first image as a retrieval result corresponding to the image to be retrieved.

[0114] Compare the semantic distance between the semantic features of the image to be retrieved and the semantic features of each first image in the first gallery with the target semantic distance of the corresponding first image. When the semantic distance between the image to be retrieved and a first image is less than the target semantic distance corresponding to the first image, it can be considered that the first image is the same as the image to be retrieved, and the first image belongs to the retrieval result of the image to be retrieved; when the semantic distance between the image to be retrieved and a first image is greater than the target semantic distance corresponding to the first image, it represents that the semantic content of the first image is different from that of the image to be retrieved, and the first image does not belong to the retrieval result of the image to be retrieved. When the semantic distance between the image to be retrieved and a first image is equal to the target semantic distance corresponding to the first image, the first image can be used as a retrieval result or not, which can be set according to the actual needs of the user.

[0115] For example, the first image in the first gallery includes image 1, image 2, image 3 and image 4, and the target semantic distances of image 1, image 2, image 3 and image 4 are respectively: threshold 1, threshold 2, threshold 3 and threshold 4. The semantic distance between the image to be retrieved and image 1 is d1, d1>threshold 1; the semantic distance between the image to be retrieved and image 2 is d2, d2>threshold 2; the semantic distance between the image to be retrieved and image 3 is d3, d3<threshold 3; the semantic distance between the image to be retrieved and image 4 is d4, d4>threshold 4; then the retrieval result of the image to be retrieved is image 3.

[0116] In practical applications, the target semantic distance is used as a criterion for judging whether the corresponding first image is similar to the image to be retrieved, and the target semantic distance may be referred to as a matching distance threshold.

[0117] The image processing method provided by the embodiment of the present invention determines the target semantic distance corresponding to each first image for distinguishing each first image from the second image as a whole based on the semantic features of each first image in the first gallery extracted by the second neural network model and the semantic features of each second image in the second gallery, and uses the determined target semantic distance as the judgment standard of similarity, searches the first image identical to the image to be retrieved from the first gallery as the retrieval result, improves the accuracy and efficiency of image retrieval, and obtains the semantic label of each sample image by clustering the semantic features of each sample image extracted by the first neural network model, trains the second neural network model based on each sample image and the corresponding semantic label, performs supervised training on the second neural network model, improves the expression ability of the semantic features extracted by the second neural network model, and further improves the accuracy and efficiency of image retrieval.

[0118] In some embodiments, when determining the target semantic distance corresponding to the first image in step S305, the target semantic distance for distinguishing the first image from the second gallery is determined based on semantic features extracted from the first image and the second image, including: determining the semantic distance between the semantic features of the first image and the semantic features of the second images in the second gallery, and sorting the semantic distances; determining the semantic distance in a set order in the sorting as the target semantic distance for distinguishing the first image from the second gallery as a whole.

[0119] Here, the semantic features of the first image and the semantic features of the second image can be mapped to the same feature space, and the feature space can be a two-dimensional space or a three-dimensional space. Figure 4 Taking the two-dimensional space shown in FIG. 4 as an example, the distribution of the semantic features of the first image and the semantic features of the second image in the feature space 40 is illustrated. The first image includes image 1, image 2, and image 3. The distribution of the semantic features F11 of image 1, the semantic features F12 of image 2, and the semantic features F13 of image 3 in the feature space 40 is shown in FIG. Figure 4 As shown, in Figure 4 In the feature space 40, F12, F13 and other points other than F13 represent semantic features of different second images in the second gallery.

[0120] After mapping the semantic features of the first image and the semantic features of the second image to the same feature space, for each first image, the semantic distance between the semantic features of the first image and the semantic features of different second images in the second gallery is calculated respectively, the semantic distances between the first image and the different second images are sorted, and the semantic distances in the set order in the sorting are selected as the target semantic distance corresponding to the first image.

[0121] In some embodiments, a target distinction degree between the first image and the second gallery is determined; and the setting order is determined according to the target distinction degree.

[0122] Here, for each first image, the target differentiation degree between the first image and the second image gallery can be determined according to the semantic distance between the first image and different second images in the second gallery. Here, the target differentiation degree represents the differentiation degree between the first image and the second gallery, and the correlation coefficient between the first image and the second gallery can be calculated by using point biserial correlation, biserial correlation method, four-point correlation method, etc. to represent the target differentiation degree between the first image and the second gallery.

[0123] Here, different target distinction levels correspond to different set orders, so that according to the target distinction level, the semantic distance in the set order in the above-mentioned semantic distance ranking is determined as the target semantic distance.

[0124] In some embodiments, when the sorting is in ascending order, the target differentiation degree is positively correlated with the set order.

[0125] Here, when the sorting is in ascending order, the higher the target distinction degree is, the better the distinction performance between the first image and the second gallery is, and the higher order semantic distance in the ascending order is selected as the target semantic distance. That is to say, the higher the target distinction degree is, the smaller the selected target semantic distance is, so that the first image can be distinguished from as many second images as possible in the second gallery.

[0126] In an embodiment of the present invention, according to the target distinction degree between the first image and the second gallery, a semantic distance of a set order is selected from the semantic features between each second image in the first image and the second gallery as the target semantic distance corresponding to the first image, so that the target distinction degree in the first image and the second gallery is reflected by the target semantic distance. The greater the target distinction degree between the first image and the second gallery, the better the performance of distinguishing the first image from the second gallery, and the better the performance of distinguishing the first image from the second image, and the performance of distinguishing the first image from the second gallery is reflected by the target semantic distance. The smaller the target semantic distance, the more second images in the second gallery can be distinguished from the first image. When performing image retrieval, the target semantic distance can be used to accurately determine whether the first image is the same as the image to be retrieved.

[0127] See also Figure 5 , Figure 5 is a flowchart of an image processing method provided by an embodiment of the present invention, which will be combined with Figure 5 The steps shown are explained.

[0128] S501: Compare the semantic distance between a first image in a first gallery and a second image in a second gallery to determine a target semantic distance for distinguishing the first image from the second gallery.

[0129] Here, the image retrieval system trains a second neural network model through each sample image and the semantic label corresponding to each sample image, and extracts corresponding semantic features from each first image in the first gallery and each second image in the second gallery through the second neural network model; based on the semantic features extracted from each first image in the first gallery and each second image in the second gallery, determines the target semantic distance for distinguishing the first image from the second gallery as a whole.

[0130] The description of S501 can refer to the description of S304 and S305. It should be noted that the semantic label corresponding to the sample image in S501 can be determined through S301 and S302, or can be determined through other methods such as manual annotation.

[0131] In some embodiments, the semantic features of each first image in the first gallery are unrelated to the semantic features of each second image in the second gallery.

[0132] S502: Retrieve a first image that is the same as the received image to be retrieved from the first gallery according to the target semantic distance.

[0133] For the description of S502 , refer to the description of S306 .

[0134] S503: Output the search result corresponding to the image to be searched.

[0135] When the server retrieves the first image identical to the image to be retrieved from the first gallery as the retrieval result of the image to be retrieved, the retrieval result can be output to the terminal and displayed to the user by the terminal.

[0136] When the terminal retrieves the first image that is the same as the image to be retrieved from the first gallery as the retrieval result of the image to be retrieved, the retrieval result can be output to the display interface to be displayed to the user.

[0137] The image processing method provided by the embodiment of the present invention determines the target semantic distance corresponding to each first image for distinguishing each first image from the second image as a whole based on the semantic features of each first image in the first gallery and the semantic features of each second image in the second gallery, and uses the determined semantic distance as a judgment standard for similarity, searches the first gallery for the same first image as the image to be retrieved as a retrieval result, thereby improving the accuracy and efficiency of image retrieval.

[0138] The image processing method provided by the embodiment of the present invention is further described below through a plurality of different retrieval scenarios.

[0139] Retrieval scene 1

[0140] In the embodiment of the present invention, image retrieval includes: retrieval based on the message digest algorithm (MD5) value of the image, image retrieval based on image pixels, and image retrieval based on semantic content (CBIR) tasks. Different from the image retrieval algorithm based on image pixels or MD5, the CBIR algorithm focuses on the semantic content of the image as a whole. For example, the MD5 values ​​of two dogs of different breeds are obviously different, and the similarity between the two at the pixel level is also low.

[0141] Among them, the image pixels of the image can be extracted based on the artificially designed image feature extractor. The image similarity algorithm used by the artificially designed image feature extractor can be Perceptual Hash (PH ash), Histogram of Oriented Gradients (HOG), Scale Invariant Feature Transform (SIFT), Discrete Cosine Transform (DCT) and other algorithms. These image similarity algorithms are not suitable for solving the semantic content matching of CBIR. Among them, the PHash algorithm or the HOG algorithm is easily affected by pixel interference. If filters or simple strokes are added to the image, the matching success rate of the algorithm will drop significantly. Figure 6 As shown, when a simple stroke is added to image 61 to obtain image 62, image 61 and image 62 will fail to match. The SIFT algorithm requires sufficient texture information of the image, otherwise the constructed 128-dimensional feature vector is not distinguishable, resulting in algorithm failure.

[0142] Retrieval scene 2

[0143] In the embodiment of the present invention, in the image retrieval algorithm based on the VGG neural network, Figure 7 The fully connected (FC) layer features 701 or the convolutional layer (FC) features 702 of the Convolutional Neural Networks (CNN) model shown are used as semantic features for image retrieval.

[0144] The fully connected layer feature 701 includes high-level features of the image 703, and these high-level image features can be well applied to retrieval tasks based on image content. The convolutional layer feature 702 in the CNN model includes the spatial position information of the image 703, and its feature expression ability is stronger than that of the fully connected layer feature. Therefore, during image retrieval, after extracting the convolutional layer features with feature dimensions of C*H*W, the convolutional layer features can be reduced to C*1*1 using sum pooling, and then the principal component analysis and whitening are used to perform post-processing operations on the reduced-dimensional features, thereby achieving the purpose of reducing feature dimensions and eliminating the correlation between feature dimensions.

[0145] The deep learning-based algorithm uses the model as the image feature extraction model. Compared with the manually designed image feature extractor, the quality of image features extracted by the CNN model is greatly improved, and the retrieval effect is significantly better than that of the manually designed image feature extractor.

[0146] However, the image retrieval algorithm based on deep neural network completes the feature extraction of the image through the convolution layer or fully connected layer of the CNN model. The quality and robustness of the features are significantly higher than those extracted manually, but it does not introduce the supervised learning process. It has obvious limitations in specific data scenarios and the retrieval effect is poor. Figure 8 As shown, when the image to be retrieved is image 81, it is expected to match image 82 as the retrieval result, but based on the fact that the pixels of image 81 and image 83 have the same texture, the actual matching result is image 83.

[0147] Retrieval scene 3

[0148] In the embodiment of the present invention, an image retrieval algorithm based on the attention mechanism is also provided, which can better utilize the spatial information of the image convolution layer features. However, the parameters of the attention network layer need supervised training. When the data set is not labeled with classification labels or the label quality is poor, the quality of the features extracted by the algorithm will be significantly reduced. Fig. 9As shown, an image retrieval system is designed based on deep neural networks and manifold ranking, and image 901 is input into CNN904 including five convolutional layers C1 to C5 and three FC layers FC6, FC7, and FC8. Image features 902 of the fully connected layer in the network are extracted through supervised learning of image 901 by CNN904; secondly, after normalizing image features 902, image features 905 extracted from images in image data set 903 by CNN904 are popularly ranked using efficient manifold ranking (EMR) algorithm to obtain popular ranking results 906, and the results returned by image features 902 in popular ranking results 906 are sorted, and the most similar image 907 is returned to the user according to the sorted results. Fig. 9 The image retrieval system shown avoids using a single similarity measurement method, such as cosine distance, Euclidean distance, etc., and adopts a manifold ranking based on manifold distance to more realistically measure the similarity between images.

[0149] but Fig. 9 The image retrieval system shown has low matching accuracy and only returns suspected similar images to the user. The user needs to judge whether there are similar images in the retrieval results, and it is not possible to accurately match the image content. The image matching system that matches the image content does not have an interactive process of manually determining whether the returned results are similar. When matching images, if the queried image contains corresponding similar images in the searched gallery, similar images will be returned. If there are no similar images (that is, it is determined that the matching criteria are not met), no results will be returned. Returning wrong results or missing results will bring a poor experience to users and even cause legal disputes. For example, the same trademark or similar trademarks may be registered repeatedly due to matching accuracy issues during the trademark registration process.

[0150] In the above-mentioned retrieval scenarios provided by the embodiments of the present invention, in the image similarity matching process, a manually designed image feature extractor or CNN model is first used to extract a feature vector expression of several hundred to several thousand dimensions from the query image, i.e., the image to be retrieved. Then, the distance between the feature vectors extracted from the query image and the seed image in the seed library is calculated using the Euclidean distance or the cosine distance. If the distance is less than the manually set threshold, the match is considered successful, otherwise it fails. The manually set threshold is an absolute threshold, and therefore, the matching result returned based on the manually set threshold is not accurate enough.

[0151] In response to the problems existing in the above-mentioned retrieval scenarios, an embodiment of the present invention proposes an image processing method, which uses a complex CNN model to assist a lightweight CNN model in supervised learning to improve the quality of feature expression, and uses the second image of the second gallery to perform a white library cleaning operation on the first image of the first gallery. The similarity between the images is measured by the feature distribution of the first image and the second image in the feature space of the lightweight CNN model, which can achieve efficient and high-precision image semantic content similarity matching.

[0152] Next, combine Fig.10 The exemplary architecture shown in the figure further describes the image processing method provided by the embodiment of the present invention. Fig.10 As shown, the image processing method provided by the embodiment of the present invention includes:

[0153] The semantic features 1003 of each image in the scene image set 1001 are extracted by the complex CNN model 1002, and the extracted semantic features are clustered and analyzed by the clustering algorithm to construct a data set 1004 with classification labels. The lightweight neural network model 1005 is trained by the data set 1004 with classification labels, and the trained lightweight neural network model 1005 extracts the semantic features 10081 of the seed image in the seed library 1006, and extracts the semantic features 10082 of the second image in the white library 1007. The semantic features 10081 of the seed image are rinsed with the white library by the semantic features 10082 of the second image, and the matching distance thresholds 1009 of various sub-images are obtained. When receiving the image 1010 to be retrieved input by the user, the semantic feature 1011 of the image 1010 is extracted, the semantic distance 1012 between the semantic feature 1011 and the semantic feature of the seed image is determined, and the matching distance threshold 1009 corresponding to the various sub-images is queried based on the semantic distance 1012 between the image 1010 and the various sub-images, and the matching result 1013, i.e., the retrieval result of the image 1010 is obtained. A further description is given below.

[0154] The first step is to generate a labeled data set.

[0155] The semantic features of each image in the scene image set are extracted through a complex neural network model, i.e., the first neural network model, and the extracted semantic features are clustered and analyzed using a clustering algorithm to construct a data set with classification labels. The complex neural network model may include: ResNet152, DenseNet164, Google Net and other neural network models with deep network depth. The clustering algorithm may be KMeans, spectral clustering, Power Iteration Clustering and other algorithms. Here, the unlabeled image features can be clustered by a clustering algorithm to construct a labeled data set.

[0156] The second step is to use the constructed data set to be labeled to train the lightweight neural network model.

[0157] The second neural network model is a lightweight neural network model, which can be MobileNet, ShuffleNet, etc. based on Bottleneck structure and group convolution structure.

[0158] In order to further improve the expression quality of image features of the online lightweight neural network model, i.e., the second neural network model, the lightweight neural network model may be trained based on a center loss function.

[0159] Although the complex neural network model has a huge amount of computation and redundant parameters, its image feature expression ability and robustness are stronger than those of the lightweight neural network model. Here, the lightweight neural network model is trained through the semantic features output by the complex neural network model to improve the expression ability of the image features of the lightweight neural network model.

[0160] The third step is to extract the semantic features of the images in the first gallery and the second gallery based on the lightweight neural network model.

[0161] After the lightweight neural network model is trained, the semantic features of the first image in the seed library and the second image in the white library are extracted based on the lightweight neural network model to obtain the semantic features of the first image and the semantic features of the second image. Here, a similarity retrieval system such as Faiss can be used to build a fast indexing system to ensure the retrieval efficiency of image features.

[0162] The fourth step is to determine the corresponding matching distance threshold by performing a white library cleaning operation on the semantic features of each first image.

[0163] An example of determining the first image matching distance threshold, i.e., the target semantic distance, in the first image library through the white library can be as follows: Figure 4 Since the second image is an image in the application scenario that is not related to the semantic content of the seed image in the seed library, it can be considered that such images do not need to be matched. Then the distance between the semantic features of each seed image in the seed library and the semantic features of all second images is calculated in turn, and then the distance to the second image closest to the current seed image is taken as the matching distance threshold of the seed image, where N is a small natural number, such as 1, 5, 10, etc.

[0164] Here, based on the fourth step, high accuracy of image retrieval can be guaranteed.

[0165] Step 5: Perform image retrieval based on the image input by the user.

[0166] In the online system acting as a client, after the semantic features of the image input by the user are extracted, when the semantic distance between the semantic features of the image input by the user and the semantic features of a seed image in the seed library is less than the matching distance threshold corresponding to the seed image, the current seed image belongs to the retrieval result matching the image input by the user and is returned and displayed to the user.

[0167] Since the number of images in the white library can be set to hundreds of thousands to millions (denoted as M), the theoretical mismatch rate of the image processing method provided by the embodiment of the present invention is: N / M. This mismatch rate is very small, and in most cases it is only a few ten-thousandths, which fully meets the commercial standards.

[0168] The image processing method provided by the embodiment of the present invention can be used as a high-precision image semantic content similarity matching method based on convolutional neural network. First, a complex neural network model is used to extract the global semantic features of the image in the application scenario; then, a clustering algorithm is used to construct a data set that can be used for classification tasks based on the extracted global semantic features, each data set corresponds to a different pseudo-label, and a lightweight neural network model is trained based on the classified data set and the corresponding pseudo-label; finally, a data set that is unrelated to the semantic content in the seed image retrieval library in the application scenario (the pixel content may be related) is collected manually or by crawlers, and the semantic features of the images in the white library and the seed library are extracted based on the lightweight neural network model, so as to determine the matching distance threshold corresponding to each seed image in the seed library.

[0169] Here, the data set consisting of data that is not related to the semantic content of the seed image in the retrieval library is defined as a "clean dataset", and the process of determining the matching distance threshold of a single seed image in the retrieval library can be called a clean dataset operation. In order to reduce the impact of the distribution differences between data sets, the semantic features extracted by the complex neural network model can be used as a basis for fine-tuning the lightweight neural network model, ensuring the quality of the global semantic features extracted by the lightweight neural network model while taking into account the efficiency of the algorithm, and then combining the clean dataset of the seed image to greatly improve the accuracy of the image semantic similarity matching results.

[0170] In practical applications, semantic content similarity, pixel level similarity and MD5 matching are inclusive relationships. Therefore, the image processing method provided by the embodiment of the present invention can be used in image semantic content retrieval and matching systems, and has the ability to cover image pixel level retrieval systems.

[0171] The image processing method provided by the embodiment of the present invention can improve the ability of the semantic feature extractor, ensure that the semantic feature expression of the image contains more discriminative information, and better measure the similarity of the image semantics based on the objective image semantic similarity evaluation (SSE) index, and ensure that the distance measurement between different images can truly reflect the distribution of the data, so as to accurately and efficiently match the user input image with the expected similar image samples in the seed library, and completely solve the current problem of low precision of image retrieval based on semantic content.

[0172] In order to better illustrate the retrieval effect of the image processing method provided by the embodiment of the present invention, the embodiment of the present invention compares the accuracy and coverage of image retrieval system 1 and image retrieval system 2, and the comparison results are shown in Table 1. Image retrieval system 1 is an image retrieval system using the image processing method of the embodiment of the present invention, and image retrieval system 2 is an image retrieval system using the ImageNet pre-training model. Here, in Table 1, the hit threshold of retrieval system 1 is the target semantic distance when N is 5, 10, and 20, respectively, and the hit threshold of retrieval system 2 is 80 points. Among them, the hit threshold is a criterion for judging whether the first image in the first gallery is a retrieval result. When the semantic distance between the first image and the image to be retrieved is less than the hit threshold, the first image belongs to the retrieval result. When the semantic distance between the first image and the image to be retrieved is greater than the hit threshold, the first image does not belong to the retrieval result.

[0173] Table 1. Example of comparison of retrieval performance between image retrieval system 1 and image retrieval system 2

[0174]

[0175] Based on the comparison results shown in Table 1, it can be shown that the accuracy and coverage of the retrieval results of the image processing method provided by the embodiment of the present invention are higher than those of the detection system 2 using the absolute hit threshold.

[0176] Fig.11 The image processing device 1100 shown in FIG. 1 is an exemplary functional structure of the image processing method provided by an embodiment of the present invention. The hardware layer of the image processing device 1100 may be Figure 1 The image processing system 100 is implemented in a terminal or server.

[0177] See also Fig.11 , Fig.111 is an exemplary structural diagram of an image processing device 1100 provided by an embodiment of the present invention, including at least one processor 1101, a memory 1102, at least one network interface 1103 and a user interface 1104. The various components in the image processing device 1100 are coupled together via a bus system 1105. It can be understood that the bus system 1105 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1105 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 1105 is not shown in FIG. Fig.11 Various buses are labeled as bus system 1105. User interface 1104 may include a display, keyboard, mouse, trackball, click wheel, keys, buttons, touch pad or touch screen, etc. Memory 1102 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0178] The memory 1102 in the embodiment of the present invention can store data to support the operation of the image processing device 1100. Examples of such data include: any computer program for operating on the image processing device 1100, such as an operating system and an application program. The operating system includes various system programs, such as a first neural network model, a second neural network model, a first image library, a second image library, etc., which are used to implement various basic services and process hardware-based tasks.

[0179] As an example of the method provided by the embodiment of the present invention being implemented by a combination of software and hardware, the image processing method provided by the embodiment of the present invention can be directly embodied as a combination of software modules executed by the processor 1101. The software module can be located in a computer-readable storage medium, and the computer-readable storage medium is located in the memory 1102. The processor 1101 reads the executable instructions included in the software module in the memory 1102, and completes the image processing method provided by the embodiment of the present invention in combination with the necessary hardware (for example, including the processor 1101 and other components connected to the bus 1105).

[0180] Combine the following Fig.11 An example of a combination of software modules in the memory 1102 is shown, including:

[0181] A first extraction unit 1110, configured to extract corresponding semantic features from a sample image through a first neural network model;

[0182] A clustering unit 1111 is used to cluster the semantic features extracted from the sample image to obtain a semantic label corresponding to the sample image;

[0183] A training unit 1112, configured to train a second neural network model using the sample images and corresponding semantic labels;

[0184] A second extraction unit 1113 is used to extract corresponding semantic features from the first image in the first gallery and from the second image in the second gallery by using the second neural network model;

[0185] A distinguishing unit 1114, configured to determine a target semantic distance for distinguishing the first image from the second image gallery according to semantic features extracted from the first image and the second image;

[0186] The retrieval unit 1115 is configured to retrieve, from the first image library, a first image that is identical to the received image to be retrieved according to the target semantic distance.

[0187] In some embodiments, the complexity of the first neural network model is greater than the complexity of the second neural network model.

[0188] In some embodiments, the semantic features of each first image in the first gallery are unrelated to the semantic features of each second image in the second gallery.

[0189] In some embodiments, the distinguishing unit 1114 is further configured to:

[0190] Determine a semantic distance between a semantic feature of the first image and a semantic feature of a second image in the second gallery, and sort the semantic distances;

[0191] The semantic distances in a set order in the ranking are determined to be the target semantic distances that distinguish the first image from the second gallery as a whole.

[0192] In some embodiments, the distinguishing unit 1114 is further configured to:

[0193] Determining a degree of target differentiation between the first image and the second image gallery;

[0194] The setting order is determined according to the target differentiation degree; wherein, when the sorting is in ascending order, the target differentiation degree is positively correlated with the setting order.

[0195] In some embodiments, the retrieval unit 1115 is further configured to:

[0196] Determining a semantic distance between the received image to be retrieved and the first image in the first gallery;

[0197] When the semantic distance is less than the target semantic distance of the first image, the corresponding first image is identified as a retrieval result corresponding to the image to be retrieved.

[0198] In some embodiments, the clustering unit 1111 is further configured to:

[0199] Mapping the semantic features extracted from the sample image to a feature space;

[0200] According to the distances between the semantic features of the sample images and different cluster centers in the feature space, the semantic features of the sample images are divided into sets corresponding to different cluster centers;

[0201] The sample images corresponding to the semantic features in the set of the same cluster center are uniformly assigned corresponding semantic labels.

[0202] In some embodiments, the training unit 1112 is further configured to:

[0203] According to the samples and the corresponding semantic labels, the parameters of the second neural network model are updated until the loss function of the second neural network model converges.

[0204] Fig.12 The image processing device 1200 shown in FIG. 1 is an exemplary functional structure of the image processing method provided by an embodiment of the present invention. The hardware layer of the image processing device 1200 may be Figure 2 The method is implemented in the terminal or server manner shown.

[0205] See also Fig.12 , Fig.12 1 is an exemplary structural diagram of an image processing device 1200 provided by an embodiment of the present invention, including at least one processor 1201, a memory 1202, at least one network interface 1203 and a user interface 1204. The various components in the image retrieval device 1200 are coupled together through a bus system 1205. It can be understood that the bus system 1205 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1205 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.12 Various buses are labeled as bus system 1205. User interface 1204 may include a display, keyboard, mouse, trackball, click wheel, keys, buttons, touch pad or touch screen, etc. Memory 1202 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0206] The memory 1202 in the embodiment of the present invention can store data to support the operation of the image processing device 1200. Examples of such data include: any computer program for operating on the image processing device 1200, such as an operating system and an application program. The operating system includes various system programs, such as a first neural network model, a second neural network model, a first image library, a second image library, etc., which are used to implement various basic services and process hardware-based tasks.

[0207] As an example of the method provided by the embodiment of the present invention being implemented by a combination of software and hardware, the image processing method provided by the embodiment of the present invention can be directly embodied as a combination of software modules executed by the processor 1201. The software module can be located in a computer-readable storage medium, and the computer-readable storage medium is located in the memory 1202. The processor 1201 reads the executable instructions included in the software module in the memory 1202, and completes the image processing method provided by the embodiment of the present invention in combination with the necessary hardware (for example, including the processor 1201 and other components connected to the bus 1205).

[0208] Combine the following Fig.12 An example of the combination of software modules in the memory 1202 is shown, including:

[0209] A determination unit 1210, configured to compare a semantic distance between a first image in a first gallery and a second image in a second gallery to determine a target semantic distance for distinguishing the first image from the second gallery;

[0210] A matching unit 1211 is used to retrieve a first image that is the same as the received image to be retrieved from the first image library according to the target semantic distance;

[0211] The output unit 1212 is used to output the search result corresponding to the image to be searched.

[0212] An embodiment of the present invention further provides a storage medium, namely, a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the image processing method provided by the embodiment of the present invention is implemented.

[0213] In summary, based on the semantic features of each first image in the first gallery and the semantic features of each second image in the second gallery, information that can distinguish the first image from the second gallery as a whole, that is, the target semantic distance, is obtained. Therefore, based on the semantic distance as the standard, the first image with the same semantic features as the image to be retrieved can be searched from the first gallery as the retrieval result, thereby achieving good accuracy and efficiency of image retrieval.

[0214] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Extracting corresponding semantic features from the sample image through the first neural network model; Clustering the semantic features extracted from the sample images to obtain semantic labels corresponding to the sample images; Training a second neural network model using the sample images and corresponding semantic labels, wherein the complexity of the first neural network model is greater than the complexity of the second neural network model; Extracting corresponding semantic features from a first image in the first gallery and a second image in the second gallery by the second neural network model, wherein the first image is an image similar to or identical to the received image to be retrieved, and the second image is an image having different semantic content from the first image; Determining a target semantic distance for distinguishing the first image from the second image gallery according to semantic features extracted from the first image and the second image; A first image that is the same as the received image to be retrieved is retrieved from the first gallery according to the target semantic distance.

2. The method according to claim 1, characterized in that The step of determining a target semantic distance for distinguishing the first image from the second image gallery according to the semantic features extracted from the first image and the second image includes: Determining a semantic distance between a semantic feature of the first image and a semantic feature of a second image in the second gallery, and sorting the semantic distances; The semantic distances in a set order in the ranking are determined to be target semantic distances that distinguish the first image from the second gallery as a whole.

3. The method according to claim 2, characterized in that The method further comprises: Determining a degree of target differentiation between the first image and the second image gallery; The setting order is determined according to the target differentiation degree; wherein, when the sorting is in ascending order, the target differentiation degree is positively correlated with the setting order.

4. The method according to claim 1, characterized in that: The retrieving, from the first gallery according to the target semantic distance, a first image identical to the received image to be retrieved, comprises: Determining a semantic distance between the received image to be retrieved and the first image in the first gallery; When the semantic distance is smaller than the target semantic distance of the first image, the corresponding first image is identified as a retrieval result corresponding to the received image to be retrieved.

5. The method according to claim 1, characterized in that The clustering of the semantic features extracted from the sample images to obtain semantic labels corresponding to the sample images includes: Mapping the semantic features extracted from the sample image to a feature space; According to the distances between the semantic features of the sample images and different cluster centers in the feature space, the semantic features of the sample images are divided into sets corresponding to different cluster centers; The sample images corresponding to the semantic features in the set of the same cluster center are uniformly assigned corresponding semantic labels.

6. The method according to claim 1, characterized in that The step of training the second neural network model using the sample images and corresponding semantic labels includes: According to the sample image and the corresponding semantic label, the parameters of the second neural network model are updated until the loss function of the second neural network model converges.

7. An image processing device, characterized in that: The device comprises: A first extraction unit, used to extract corresponding semantic features from the sample image through a first neural network model; A clustering unit, used for clustering the semantic features extracted from the sample image to obtain a semantic label corresponding to the sample image; A training unit, configured to train a second neural network model using the sample images and corresponding semantic labels, wherein the complexity of the first neural network model is greater than the complexity of the second neural network model; a second extraction unit, configured to extract corresponding semantic features from a first image in the first gallery and from a second image in the second gallery by using the second neural network model, wherein the first image is an image similar to or identical to the received image to be retrieved, and the second image is an image having a different semantic content from that of the first image; a distinguishing unit, configured to determine a target semantic distance for distinguishing the first image from the second image gallery based on semantic features extracted from the first image and the second image; A retrieval unit is used to retrieve a first image that is the same as the received image to be retrieved from the first image library according to the target semantic distance.

8. The device according to claim 7, characterized in that The distinguishing unit is further used for: Determining a semantic distance between a semantic feature of the first image and a semantic feature of a second image in the second gallery, and sorting the semantic distances; The semantic distances in a set order in the ranking are determined to be target semantic distances that distinguish the first image from the second gallery as a whole.

9. The device according to claim 8, characterized in that The distinguishing unit is further used for: Determining a degree of target differentiation between the first image and the second image gallery; The setting order is determined according to the target differentiation degree; wherein, when the sorting is in ascending order, the target differentiation degree is positively correlated with the setting order.

10. The device according to claim 7, characterized in that The clustering unit is further used for: Mapping the semantic features extracted from the sample image to a feature space; According to the distances between the semantic features of the sample images and different cluster centers in the feature space, the semantic features of the sample images are divided into sets corresponding to different cluster centers; The sample images corresponding to the semantic features in the set of the same cluster center are uniformly assigned corresponding semantic labels.

11. An image processing device, characterized in that: The device comprises: a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to implement the steps of the image processing method according to any one of claims 1 to 6 when executing the computer program stored in the memory.

12. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images