Image retrieval model training method, image retrieval method and device

Abstract

The embodiments of the present application disclose an image retrieval model training method, an image retrieval method and an apparatus, wherein the training method comprises: obtaining a training image set including a sample image and a similar sample of the sample image; performing defect creation processing on the training image set to obtain a defect image set, dividing the defect image set, and performing structural destruction processing on the first defect image set obtained by the division to obtain a damaged image set; inputting the damaged image set and the second defect image set into a neural network model to obtain a feature vector of the target image set; constructing a classifier and a damage identifier, determining a classification loss function according to the feature vector and the classification loss of the classifier, and determining a damage loss function according to the feature vector and the damage recognition loss of the damage identifier; optimizing the neural network model according to the classification loss function and the damage loss function to obtain an image retrieval model, and enhancing the recognition capability of similar images through the image retrieval model.

Description

Technical Field

[0001] The present application relates to an image retrieval technology and provides an image retrieval model training method, an image retrieval method and a device. Background Art

[0002] With the development of multimedia information, people have more and more demands for image retrieval, and the accuracy of image retrieval is also getting higher and higher. Image retrieval technology refers to retrieving images that meet the requirements from a massive database based on input information. The related technology is generally based on global features. This retrieval technology can distinguish images with huge differences, but cannot distinguish very similar images, so it cannot accurately distinguish similar images. Summary of the invention

[0003] The purpose of this application is to provide an image retrieval model training method, image retrieval method, device, equipment and storage medium, starting from the training data structure, network structure and other levels, deeply considering the impact of various factors on the model's representation ability, allowing the model to learn more detailed information, thereby enhancing the ability to recognize similar images.

[0004] The present application proposes an image retrieval model training method, comprising: obtaining a training image set, wherein the training image set comprises a sample image and a similar sample of the sample image; performing defect creation processing on the training image set to obtain a defect image set, and dividing the defect image set into a first defect image set and a second defect image set, and performing structural destruction processing on the first defect image set to obtain a damaged image set; using the damaged image set and the second defect image set as target image sets, and inputting the target image set into an initial convolutional neural network model to obtain a feature vector of the target image set; constructing a classifier and a damage identifier, determining a classification loss function of the initial convolutional neural network model according to the feature vector and the classification loss of the classifier, and determining a damage loss function of the initial convolutional neural network model according to the feature vector and the damage recognition loss of the damage identifier; optimizing the initial convolutional neural network model according to the classification loss function and the damage loss function to obtain an image retrieval model.

[0005] Furthermore, the defect creation processing performed on the training image set to obtain the defect image set includes: setting each mask area on the image in the training image set; removing the image content in each mask area, or replacing the image content in each mask area with image noise to obtain the defect image set with partial image information masked out.

[0006] Furthermore, the structural destruction processing of the first defective image set to obtain a damaged image set includes: dividing the images of the first defective image set into a plurality of small blocks; generating structurally damaged images after disrupting the plurality of small blocks to obtain the damaged image set, and retaining the damage information of the images.

[0007] Furthermore, determining the classification loss function of the initial convolutional neural network model based on the feature vector and the classification loss of the classifier includes: inputting the feature vector into the classifier to obtain the probability value that the image corresponding to the feature vector belongs to the image category; determining the classification loss of the classifier based on the probability value that the image corresponding to the feature vector belongs to the image category and the true image category to which the image corresponding to the feature vector belongs; and combining the probability value with the classification loss of the classifier to obtain the classification loss function of the initial convolutional neural network model.

[0008] Furthermore, the damage loss function of the initial convolutional neural network model is determined based on the feature vector and the damage recognition loss of the damage identifier, including: inputting the feature vector into the damage identifier to obtain a probability value of whether the image corresponding to the feature vector has been damaged; determining the damage recognition loss of the damage identifier based on the probability value of whether the image corresponding to the feature vector has been damaged and the damage information corresponding to the image corresponding to the feature vector; combining the probability value of whether the image corresponding to the feature vector has been damaged with the damage recognition loss to obtain the damage loss function of the initial convolutional neural network model.

[0009] Furthermore, the initial convolutional neural network model is a residual network ResNet; the initial convolutional neural network model is a residual network ResNet; before inputting the target image set into the initial convolutional neural network model to obtain the feature vector of the target image set, the method also: adjusts the network structure of the ResNet to change the downsampling operation in the ResNet from 32 times to 16 times.

[0010] Furthermore, the sample image is a trademark image, and obtaining the training image set includes: obtaining a registered trademark image, a counterfeit trademark image of the registered trademark image, and an icon image similar to the registered trademark image; and using the registered trademark image, the counterfeit trademark image, and the icon image as the training image set.

[0011] The present application also proposes an image retrieval method, which is based on an image retrieval model trained by the image retrieval model training method as described above, and the method includes: obtaining an image to be identified, inputting the image to be identified into the image retrieval model to obtain a feature vector of the image to be identified; inputting each retrieval image into the image retrieval model to obtain a feature vector of each retrieval image, and comparing the feature vector of each retrieval image with the feature vector of the image to be identified to obtain a retrieval result of the image to be identified.

[0012] The present application also proposes an image retrieval model training device, including: an image set acquisition module, used to acquire a training image set, the training image set including a sample image and a similar sample of the sample image; a processing module, used to perform defect creation processing on the training image set to obtain a defect image set, and divide the defect image set into a first defect image set and a second defect image set, and perform structural destruction processing on the first defect image set to obtain a damaged image set; a feature acquisition module, used to use the damaged image set and the second defect image set as target image sets, and input the target image set into an initial convolutional neural network model to obtain a feature vector of the target image set; a loss determination module, used to construct a classifier and a damage identifier, determine the classification loss function of the initial convolutional neural network model according to the feature vector and the classification loss of the classifier, and determine the damage loss function of the initial convolutional neural network model according to the feature vector and the damage recognition loss of the damage identifier; a model optimization module, used to optimize the initial convolutional neural network model according to the classification loss function and the damage loss function to obtain an image retrieval model.

[0013] The present application also proposes an image retrieval device, which is based on an image retrieval model trained by the image retrieval model training device as described above, and the device includes: an image acquisition module, used to acquire an image to be identified, and input the image to be identified into the image retrieval model to obtain a feature vector of the image to be identified; an image retrieval module, used to input each retrieval image into the image retrieval model to obtain a feature vector of each retrieval image, and compare the feature vector of each retrieval image with the feature vector of the image to be identified to obtain a retrieval result of the image to be identified.

[0014] The present application also proposes an electronic device, which includes: a memory storing computer-readable instructions; and a processor reading the computer-readable instructions stored in the memory to execute the method described above.

[0015] The present application also proposes a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of an electronic device, the electronic device executes the method as described above.

[0016] Compared with the prior art, this application has the following beneficial effects:

[0017] In the technical solution provided by the present application, first, by obtaining a training image set, and the training image set includes sample images and similar samples of the sample images, the model can pay attention to the discriminative details between two similar categories in the training of the classification task, and this attention ability can be transferred to the subsequent retrieval task to a certain extent; second, the training images are processed by creating defects. After erasing part of the information of the training images, if the model wants to ensure correct classification, it will pay attention to the remaining more detailed parts, so that the trained model can pay more attention to the detail information; third, the sequential structure of the original image of the training image is destroyed to reduce the model's dependence on global information, and finally, by damaging the recognition branch and the classification branch, the model is assisted in learning the detail information and enhancing the model's recognition ability for similar images; fourth, by adjusting the network structure of the model so that the final output features retain more detail information, the trained model can more accurately distinguish similar images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of an image retrieval model training method shown in an exemplary embodiment of the present application;

[0019] Figure 2 yes Figure 1 A flowchart of step S110 in the illustrated embodiment in an exemplary embodiment;

[0020] Figure 3 yes Figure 1 A flowchart of step S120 in the illustrated embodiment in an exemplary embodiment;

[0021] Figure 4 yes Figure 1 A flowchart of step S120 in the illustrated embodiment in another exemplary embodiment;

[0022] Figure 5 yes Figure 1 A flowchart of step S140 in the illustrated embodiment in an exemplary embodiment;

[0023] Figure 6 yes Figure 1 A flowchart of step S140 in the illustrated embodiment in another exemplary embodiment;

[0024] Figure 7is a flowchart of an image retrieval method shown in an exemplary embodiment of the present application;

[0025] Figure 8 is a flowchart of another image retrieval model training method shown in an exemplary embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of a model training shown in an exemplary embodiment of the present application;

[0027] Fig.10 is a schematic diagram of an image retrieval comparison shown in an exemplary embodiment of the present application;

[0028] Fig.11 is a schematic diagram of an image retrieval model training device shown in an exemplary embodiment of the present application;

[0029] Fig.12 is a schematic diagram of an image retrieval device shown in an exemplary embodiment of the present application;

[0030] Fig.13 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0031] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0033] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0034] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0035] See also Figure 1 , Figure 1 The image retrieval model training method is shown according to an exemplary embodiment. The image retrieval model training method includes steps S110 to S150, which are described in detail as follows:

[0036] S110: Acquire a training image set, wherein the training image set includes a sample image and similar samples of the sample image.

[0037] The sample images may be various types of images, for example, the sample images may be trademark images, face images, car images, etc., wherein a certain type of images may be extracted from the image library, such as extracting various trademark images as sample images.

[0038] In order for the model to focus on the discriminative details between similar images during the training of classification tasks, in this embodiment, it is necessary to obtain similar samples of the sample image. For example, after collecting the sample image, an image similar to the sample image is supplemented, and then the sample image and the supplemented similar samples are used as a training image set; for another example, similar samples similar to the sample image are directly put into the sample set to obtain a training image set.

[0039] It should be noted that when the sample image is a trademark image, Figure 2 As shown, the similar samples for supplementing the sample image in step S110 include:

[0040] S111, acquiring a registered trademark image, a counterfeit trademark image of the registered trademark image, and an icon image similar to the registered trademark image.

[0041] In this embodiment, a registered trademark refers to a trademark that has been approved for registration by a trademark management agency in accordance with the law. The registered trademark must be an image of a genuine trademark. A counterfeit trademark refers to a trademark that imitates the same or similar trademark as the registered trademark. The registered trademark and the counterfeit trademark are extremely similar. For example, the registered trademark is "RIO" and the counterfeit trademark is "RID". The types of counterfeit trademarks of a registered trademark may reach hundreds or thousands. Therefore, the counterfeit trademark image of the registered trademark image can be used as a sample image.

[0042] It can be understood that an icon refers to an image with referential meaning and identification properties. The icon dataset includes a variety of icons for various topics. For example, for cars, it includes icons formed by multiple cars, and the shapes of multiple cars are roughly the same. Therefore, an icon image similar to a registered trademark can be used as a sample image.

[0043] S112: Use the registered trademark images, counterfeit trademark images and icon images as training image sets.

[0044] All sample registered trademark images, counterfeit trademark images and icon images similar to the sample registered trademark images are taken as training image sets.

[0045] S120, performing defect creation processing on the training image set to obtain a defect image set, dividing the defect image set into a first defect image set and a second defect image set, and performing structure destruction processing on the first defect image set to obtain a destroyed image set.

[0046] In this embodiment, defect creation processing is performed on the training image set to mask part of the image information of the training image set; the defective images are divided into a first defective image set and a second defective image set, and the first defective image set is subjected to structural destruction processing to obtain a destroyed image set, that is, structural destruction processing is performed on part of the images in the defective image set to destroy the global information of the images.

[0047] The defect manufacturing process may be performed on all images in the training image set, or on part of the images in the training image set, which is not limited here. Figure 3 As shown, the process of obtaining the defective image set in step S120 includes:

[0048] S1211. Set a mask region on an image in a training image set.

[0049] The mask area may be set on each image in the training image set, or on part of the images. It is understandable that the mask areas on different images may be different or the same. The area of ​​the mask area is smaller than the area of ​​the image. Optionally, the area of ​​the mask area does not exceed 30% of the area of ​​the image. The shape of the mask area includes but is not limited to regular shapes such as circle, square, rectangle, etc. Of course, the shape of the mask area may also be an irregular shape, such as a cloud shape.

[0050] In one example, the setting of the mask area can be a random setting, that is, the mask area is randomly set on the image; in another example, the setting of the mask area can also be set according to a rule, for example, the image is divided into multiple grids, and one of the multiple grids is set as the mask area in turn according to the order of the image.

[0051] S1212: remove the image content in each mask area, or replace the image content in each mask area with image noise, so as to obtain a defective image set with partial image information masked out.

[0052] In this embodiment, all image contents in the mask area are erased, for example, all images in the mask area are blanked, or all image contents in the mask area are replaced by image noise, wherein image noise refers to unnecessary or redundant interference information present in the image data, and the image noise may be Gaussian noise; by erasing the image content in the mask area or replacing it with image noise, a part of the image information is lost in the images in the training image set, thereby obtaining a defective image set.

[0053] It should be noted that after obtaining the defective image set, the defective image set is divided into a first defective image set and a second defective image set, which can be divided evenly, that is, the number of images in the first defective image set and the second defective image set is the same; it can also be divided proportionally, for example, the first defective image set is 70% of the defective image set, and the second defective image set is 30% of the defective image set, and then some images in the defective image set can be taken for structural destruction processing.

[0054] like Figure 4 As shown, the process of obtaining the damaged image set in step S120 includes:

[0055] S1221. Divide the images of the first defective image set into a plurality of small blocks.

[0056] S1222. After disrupting the multiple small blocks, an image with a structure destroyed is generated to obtain a destroyed image set, and the destruction information of the image is retained.

[0057] The images of the first defective image set are divided into n*n small blocks, where n can be any integer, and the division method can be random division into n*n small blocks, or uniform division into n*n small blocks. Of course, for different images in the first defective image set, the corresponding values ​​of n may also be different; then after shuffling these small blocks, the shuffled small blocks are recombined to newly generate images with structural damage, and then a damaged image set is obtained through each newly produced image, and the damage information of each image is retained, wherein the shuffling method can be random shuffling or regular shuffling, and the damage information can be used to characterize whether the image has undergone structural damage; in one example, the damage information can also be used to characterize the shuffling method of the small blocks.

[0058] In this embodiment, after structural destruction processing is performed on some images in the defective image set, a damaged image set and a second defective image set are obtained. At this time, each image in the damaged image set and the second defective image set can be labeled with a label to indicate whether it has been damaged. For example, images that have not been structurally damaged are labeled "0", and images that have been structurally damaged are labeled "1" according to the damage information.

[0059] It is worth noting that when the images in the damaged image set are trademark images, when the images in the first defective image set (i.e., trademark images) are subjected to structural destruction processing, they are divided into n*n small blocks, and the value of n is relatively small; considering that the trademark images themselves are relatively small in size and the patterns are generally not particularly complex, the value of n can be relatively small, and preferably, n is an integer within 5. Since trademark images can be graphic trademarks or text trademarks, and text trademarks have the situation of multiple words splicing, if there is a space at 1 / n of the length, then such trademark images are not suitable for structural destruction. For some graphic trademark images, if the graphics of the trademark image are still consistent before and after the destruction, such trademark images are also not suitable for structural destruction; therefore, in this embodiment, it is necessary to filter out text trademark images whose length is exactly 1 / n and whose graphics are still consistent before and after the destruction from the damaged image set, and use the filtered images as the second defective image set.

[0060] In an example of this embodiment, before performing defect creation processing on a training image set to obtain a defect image set, data enhancement processing may be performed on the training image set to increase the diversity of training samples and improve the accuracy and robustness of subsequent model training.

[0061] Among them, data enhancement processing includes: flipping, rotation, graying, color transformation and cropping. Flipping can be horizontal and vertical flipping of images in the training image set, and rotation can be random rotation of images at a certain angle, such as 90°, 180°, etc. Graying can be random graying of images, converting RGB images into grayscale values; cropping randomly samples a part from the image and then adjusts the size of this part to the image size; color transformation can change the color of the image; the enhanced image obtained by the enhancement process is added to the training image set, and then the training image set after the compensation enhanced image is subsequently defect-made to obtain a defect image set.

[0062] S130, taking the damaged image set and the second defective image set as the target image set, and inputting the target image set into the initial convolutional neural network model to obtain a feature vector of the target image set.

[0063] In this embodiment, the target image set is input into the initial convolutional neural network model, wherein the initial neural network model is a convolutional neural network composed of multiple backbones. Preferably, the initial convolutional neural network model in this embodiment is ResNet (residual network), wherein after the image is input into ResNet, ResNet downsamples the image by 32 times, and then extracts the image representation to obtain the feature vector of the image.

[0064] It is worth noting that in this embodiment, before inputting the target image set into the initial convolutional neural network model to obtain the feature vector of the target image set, the network structure of ResNet needs to be adjusted. For example, when the images in the target image set are trademark images, since the trademark images are small, it is easy to lose detail information after so many layers of downsampling. Therefore, in this embodiment, the ResNet network structure is modified, and the original 32-fold downsampling is modified to 16-fold, so that the feature vector output by ResNet can retain more detail information; the target image set is input into the adjusted ResNet network model, and the images of the target image set are downsampled 16 times through the ResNet network, and then the image representation is extracted to obtain the feature vectors of each image in the target image set, and the feature vectors of each image are merged to obtain the feature vector of the target image set.

[0065] S140, constructing a classifier and a damage identifier, determining the classification loss function of the initial convolutional neural network model according to the feature vector and the classification loss of the classifier, and determining the damage recognition loss function of the initial convolutional neural network model according to the feature vector and the damage recognition loss of the damage identifier.

[0066] In this embodiment, a classifier and a damage identifier are constructed, wherein the classifier and the damage identifier are both represented as a fully connected layer (FC) in the neural network. For example, if the extracted feature vector is N-dimensional, the classifier is an FC from N-dimensional to M-dimensional, where M and N are both positive integers; M is the number of image categories, and each vector value of the M-dimensional output vector represents the probability value that the image corresponding to the feature vector belongs to each image category; when the feature vector is N, the damage identifier is an FC from N-dimensional to 2-dimensional, and the 2-dimensional output respectively represents the probability that the image corresponding to the feature vector is damaged and the probability that it is not damaged.

[0067] In this embodiment, if Figure 5 As shown, the process of determining the classification loss function of the initial convolutional neural network model in step S140 includes:

[0068] S1411, inputting the feature vector into a classifier to obtain a probability value of the image input image category corresponding to the feature vector;

[0069] S1412, determining the classification loss of the classifier according to the probability value that the image corresponding to the feature vector belongs to the image category and the real image category to which the image corresponding to the feature vector belongs;

[0070] S1413. Combining the probability value with the loss function of the classifier to obtain the classification loss function of the initial convolutional neural network model.

[0071] It can be understood that the classifier can map the feature vector into an m-dimensional vector, and then obtain the probability value that the image corresponding to the feature vector belongs to the image category; and the classification loss of the classifier is used to measure the degree of inconsistency in the classifier's prediction. Specifically, the classification loss of the classifier can be obtained based on the probability value of the image category output by the classifier and the actual true image category of the image. For example, the input feature vector is x, the probability value output by the classifier is y=f(x), and the image corresponding to the feature vector belongs to the true image category y', then the classification loss of the classifier is L(y', f(x)).

[0072] Optionally, the classification loss of the classifier includes multiple forms, such as softmax (cross entropy loss function), triplet loss (a loss function in deep learning), cosface (Large Margin Cosine Loss for Deep Face Recognition), Arcface (Additive Angular Margin Loss, additive angular margin loss function). Preferably, Arcface can be selected as the classification loss of the classifier, and then the probability value output by the classifier is combined with the classification loss of the classifier to obtain the classification loss function loss under the current initial convolutional neural network model.

[0073] In this embodiment, if Figure 6 As shown, the process of determining the damage loss function of the initial convolutional neural network model in step S140 includes:

[0074] S1421, inputting the feature vector into a damage identifier to obtain a probability value of whether the image corresponding to the feature vector has been damaged;

[0075] S1422, determining a damage recognition loss of a damage identifier according to a probability value of whether the image corresponding to the feature vector has been damaged and damage information corresponding to the image corresponding to the feature vector;

[0076] S1423. Combining the probability value of whether the image corresponding to the feature vector has been damaged with the damage recognition loss to obtain the damage loss function of the initial convolutional neural network model.

[0077] As before, the damage identifier is an N-dimensional to 2-dimensional FC, and the 2-dimensional outputs respectively represent the probability that the image corresponding to the feature vector is damaged and the probability that it is not damaged. Therefore, after the feature vector is input into the damage identifier, the probability value of whether the image corresponding to the feature vector has been damaged can be obtained. For example, when the probability of being damaged is greater than the probability of being undamaged, the damage identifier outputs the probability that the image is damaged; the damage recognition loss of the damage identifier is used to predict the degree of inconsistency of the damage identifier. Since the damage information of the image is retained when the image is structurally damaged in advance, and the damage information can be used to characterize whether the image has been structurally damaged, the damage recognition loss of the damage identifier can be determined based on the probability value of whether the image corresponding to the feature vector has been damaged, and the damage information corresponding to the image corresponding to the feature vector. If the damage information characterizes that the image has been structurally damaged, it is represented by "1", and the damage information is the damage label.

[0078] Optionally, the damage recognition loss of the damage identifier can adopt softmax, and then the probability value of whether the image output by the damage identifier is damaged and the damage recognition loss are combined to obtain the damage loss function loss under the current initial convolutional neural network model.

[0079] S150. Optimize the initial convolutional neural network model according to the classification loss function and the damage loss function to obtain an image retrieval model.

[0080] In this embodiment, the model parameters of the initial convolutional neural network model are adjusted by using the classification loss function and the damage loss function to optimize the initial convolutional neural network model.

[0081] Optionally, obtaining the image retrieval model in step S150 includes:

[0082] The classification loss function and the damage loss function are weighted averaged to obtain the total loss function; the model parameters of the initial convolutional neural network model are optimized according to the total loss function until the total loss function converges to obtain the image retrieval model.

[0083] The total loss function includes: L = (λ1L3 + λ2L1) / 2, where λ1 and λ2 are weights, L3 is the classification loss function, and L1 is the damage loss function; λ3 and λ1 can be flexibly adjusted according to actual needs, for example, λ3 is 0.4 and λ1 is 0.6. Based on this total loss, back propagation is performed to optimize the model parameters. When the total loss function converges, it means that the parameters of the initial convolutional neural network model are optimal.

[0084] The image retrieval model training method provided in this embodiment allows the model to learn more detailed information by simply supplementing similar samples, creating defects in the image, destroying the original image sequence structure and adding a damage identifier, adjusting the network structure, etc., thereby enhancing the recognition ability of similar images; in general, starting from the data structure, network structure and other levels, in-depth consideration of the impact of various factors on the model's representation ability, using a simple and efficient way to implement the image retrieval solution, and has the advantages of simple training, high robustness and high precision.

[0085] In this embodiment, an image retrieval method is also provided. The image retrieval method is based on an image retrieval model trained according to the above-mentioned image retrieval model training method. Figure 7 As shown, the method includes:

[0086] S710, obtaining an image to be identified, and inputting the image to be identified into an image retrieval model to obtain a feature vector of the image to be identified;

[0087] S720, input each search image into the image search model to obtain a feature vector of each search image, and compare the feature vector of each search image with the feature vector in the feature library to obtain a search result of the image to be identified.

[0088] It can be understood that when an image retrieval model is obtained by training according to the training method of the image retrieval model in the above embodiment, the image retrieval model can be directly applied formally at this time. Suppose the image a to be identified is input into the image retrieval model, and the feature vector a of image a is extracted through the image retrieval model. At this time, the feature vector and image a can be stored together in the feature library; the current image database includes various known retrieval images, and each retrieval image is input into the image retrieval model to obtain the feature vector of each retrieval image, and then the feature vector of each retrieval image is compared with the feature vector a in the feature library, so that it can be determined which retrieval image the image to be identified corresponding to the feature vector is similar to.

[0089] For ease of understanding, the training method of the image retrieval model is described in this embodiment using a trademark image. Figure 8 As shown, the training method of the image retrieval model includes:

[0090] S810: Supplement similar samples in the sample trademark image data to obtain a training image set.

[0091] Since there are tens of thousands of types of trademarks and most trademarks are very individual, it is not easy to collect similar trademarks in application scenarios. This embodiment proposes two methods for quickly collecting similar trademarks: first, collect imitation trademarks in application scenarios. Generally, genuine trademarks (i.e. registered trademarks) and imitation trademarks (i.e. counterfeit trademarks) are very similar, and the number of imitation trademarks of a genuine product may reach hundreds or thousands. This type of data collection is relatively easy. Second, collect similar icons in icon data as a type of trademark data. Generally, each public icon data set will have multiple icons for a certain theme. This type of icon data can be used as similar trademarks.

[0092] S820, after general data enhancement is performed on the images in the training image set, random defects are first created, and then the image structure is randomly destroyed, and the destruction information is retained.

[0093] The collected input image set needs to go through a series of preprocessing tasks before the neural network is used, including:

[0094] First, perform random data augmentation on the input image set. The data augmentation methods can be rotation, flipping, color change, etc. The purpose of this step is to simulate real trademark scenarios and increase data diversity.

[0095] Second, random defects are created for the input image set. The specific implementation method of this step is to randomly select a rectangular frame on the original image, erase all the contents in the frame or replace all the contents in the frame with Gaussian noise. The area of ​​the rectangular frame generally does not exceed 30% of the entire image. The purpose of this step is to mask part of the information of the image. If the model wants to achieve the original classification effect, it can only pay more attention to the remaining details. Therefore, at this time, part of the global outline has been destroyed. In this way, the model output from the training will autonomously pay attention to the details of the trademark, not just the overall outline information; at the same time, such processing also simulates the trademark occlusion scene in real scenes, and has a stronger ability to recognize the occluded trademark.

[0096] Third, take a part of the data processed above for structural destruction. The specific destruction operation is as follows: randomly divide the original trademark image into small blocks of n*n, where n can be any integer, and then randomly shuffle these small blocks to generate a new damaged image, and assign a destruction label indicating whether the image has been damaged to all images. For example, assign '0' to the undamaged ones and '1' to the damaged images. There are the following characteristics during the implementation process: First, considering that the trademark images are relatively small in size and the patterns are generally not particularly complex, the value of n can be relatively small, within 5, for example, n is 3. Second, for text trademarks, due to the existence of multi-word splicing, if there is exactly a space at 1 / n of the length, then such trademarks are not suitable for structural destruction. For example, for the trademark 'MeiJi', and the damaged trademark 'JiMei', 'JiMei' may be another existing trademark. Intuitively, the trademarks before and after destruction cannot be classified into the same category, otherwise it will cause model confusion. Third, for some pattern trademarks with the same pattern before and after destruction, they are also not suitable for structural destruction. The purpose of this step of destruction is to reduce the model's dependence on global information. The global information of the damaged image is destroyed. If the model wants to achieve accurate classification, it directly relies on the detailed information of the damaged image, so it will spontaneously focus on the details. At the same time, through the subsequent damage recognizer, the automatic discrimination of whether the image has been damaged can assist in optimizing the model's ability for the classification task, and this ability can be transferred to the retrieval task to a certain extent.

[0097] S830. Construct a convolutional neural network model to obtain the abstract feature vectors of the training image set.

[0098] In this step, multiple backbones can be selected to form the convolutional neural network model. In this embodiment, ResNet is selected. However, considering that ResNet downsamples the input image by 32 times and then extracts the image representation, and trademark images are generally small, and it is easy to lose detailed information after so many layers of downsampling. Therefore, in this embodiment, the ResNet network structure is modified, and the original 32-fold downsampling is modified to 16-fold.

[0099] After this step, for the images preprocessed in S820, the corresponding feature vectors are extracted.

[0100] S840. Construct a classifier and a damage recognizer, determine the classification loss function of the convolutional neural network model according to the feature vectors and the classification loss of the classifier, and determine the damage loss function of the convolutional neural network model according to the feature vectors and the damage recognition loss of the damage recognizer.

[0101] This step includes two tasks: classification task and damage identification task, which are completed by the classifier and damage identifier respectively. The classifier maps the feature vector output by S830 into an m-dimensional vector, where m is the number of trademark categories. Each vector value represents the probability value of the image belonging to each trademark category. The probability value and the actual trademark category are calculated to obtain the classification loss of the classifier. The damage identifier maps the feature vector output by S830 into a two-dimensional vector, which respectively represents the probability value of whether the image has been damaged. The probability value and the damage label in S820 are calculated to obtain the damage identification loss of the damage identifier.

[0102] Both the classifier and the damage identifier are represented as a fully connected layer (FC) in the neural network. Assuming that the feature vector extracted in S830 is N-dimensional, the classifier is an FC from N-dimensional to m-dimensional. Each value of the m-dimensional output vector represents the possible probability of the corresponding trademark category. This value is combined with the classification loss of the classifier to obtain the classification loss under the current model parameters. The classification loss can be used in many forms. Arcface is used in this embodiment. The formula of Arcface is as follows:

[0103]

[0104] Where s is the feature normalization parameter, m is the angle penalty parameter, is the angle between the input feature Arcface loss function layer weight vector, θ j is the weight vector w j and the eigenvector x i The angle between them, Arcface is implemented in FC, so assuming x i is the feature vector output in S830, w j is the mapping vector of FC, originally w in arcface j With x i The result of multiplication is converted into the form of cosine, that is:

[0105]

[0106] The damage detector is an N-dimensional to 2-dimensional FC in the neural network. The 2-dimensional output represents the probability of damage and the probability of non-damage. This value is combined with the damage recognition loss of the damage detector to obtain the damage loss under the current model parameters. The damage recognition loss can use a simple softmaxloss:

[0107]

[0108] Where N is the number of samples, i is the i-th sample, j is the j-th category, W and b represent the weight and bias of softmax respectively, and x i represents the feature vector, y i Represents the true label of the sample.

[0109] like Fig. 9 As shown in the figure, after obtaining the training image set, random data enhancement, random defects and structural damage processing are performed, and then input into the convolutional neural network model, and the feature vector output by the model is input into the classifier and damage identifier to obtain the corresponding loss.

[0110] S850. Combining the above two losses to obtain the final total loss, and optimizing the parameters of the convolutional neural network model based on the total loss to obtain an image retrieval model.

[0111] The classification loss and damage recognition loss are weighted averaged to obtain the final total loss function, and the model parameters are optimized through back propagation based on this total loss.

[0112] After the image retrieval model is obtained, the trained model is formally applied. This model is the convolutional neural network model obtained in S830. The convolutional neural network model is applied to the retrieval system.

[0113] like Fig.10 As shown, the image A to be identified is input into the image retrieval model to obtain the feature vector A, and each known retrieval image A', B, C, D, E, etc. is input into the image retrieval model to obtain each feature vector A', B, C, D, E, etc., and then the feature vector A is compared with the feature vectors A', B, C, D, E, etc., and finally it is determined that the image A to be identified is similar to the retrieval image A'.

[0114] The technical solution provided in this embodiment supplements similar samples of trademark images to obtain a training image set, so that the model can pay attention to the discriminative details between two similar categories during the training of classification tasks. This attention ability can be transferred to subsequent retrieval tasks to a certain extent, and the trademark images are processed for defects. After erasing part of the information of the training image, if the model wants to ensure correct classification, it will pay attention to the remaining more detailed parts, so that the trained model can pay more attention to the detailed information; thirdly, the sequential structure of the original trademark image is destroyed to reduce the model's dependence on global information, and finally, the damaged recognition branch and the classification branch are used to assist the model in learning detailed information and enhance the model's recognition ability for similar trademark images. Fourthly, by adjusting the network structure of the model, the final output feature retains more detailed information, and the trained model can more accurately distinguish similar trademarks.

[0115] The following describes an apparatus embodiment of the present application, which can be used to execute the image retrieval model training method and image retrieval method in the above-mentioned embodiments of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiments of the image retrieval model training method and image retrieval method in the present application.

[0116] like Fig.11 As shown, Fig.11 : is a schematic diagram of an image retrieval model training device shown in an exemplary embodiment of the present application, and the image retrieval model training device includes:

[0117] An image set acquisition module 1110 is used to acquire a training image set, where the training image set includes a sample image and similar samples of the sample image;

[0118] The processing module 1120 is used to perform defect creation processing on the training image set to obtain a defect image set, divide the defect image set into a first defect image set and a second defect image set, and perform structure destruction processing on the first defect image set to obtain a destroyed image set;

[0119] A feature acquisition module 1130 is used to use the damaged image set and the second defective image set as a target image set, and input the target image set into an initial convolutional neural network model to obtain a feature vector of the target image set;

[0120] A loss determination module 1140 is used to construct a classifier and a damage identifier, determine a classification loss function of the initial convolutional neural network model according to the feature vector and the classification loss of the classifier, and determine a damage loss function of the initial convolutional neural network model according to the feature vector and the damage identification loss of the damage identifier;

[0121] The model optimization module 1150 is used to optimize the initial convolutional neural network model according to the classification loss function and the damage loss function to obtain an image retrieval model.

[0122] In an exemplary embodiment, the processing module 1120 includes a defect creation processing unit, which is used to set each mask area on the image in the training image set; remove the image content in each mask area, or replace the image content in each mask area with image noise to obtain a defect image set that masks part of the image information.

[0123] In an exemplary embodiment, the processing module 1120 also includes a structure destruction processing unit, which is used to divide the images of the first defective image set into multiple small blocks; after disrupting the multiple small blocks, a structurally destroyed image is generated to obtain a destroyed image set, and the destruction information of the image is retained.

[0124] Exemplarily, the loss determination module 1140 includes a classification loss function determination unit, which is used to input the feature vector into the classifier to obtain the probability value that the image corresponding to the feature vector belongs to the image category; determine the classification loss of the classifier according to the probability value that the image corresponding to the feature vector belongs to the image category and the true image category to which the image corresponding to the feature vector belongs; and combine the probability value with the classification loss of the classifier to obtain the classification loss function of the initial convolutional neural network model.

[0125] Exemplarily, the loss determination module 1140 also includes a damage loss function determination unit, which is used to input the feature vector into the damage identifier to obtain the probability value of whether the image corresponding to the feature vector has been damaged; determine the damage recognition loss of the damage identifier based on the probability value of whether the image corresponding to the feature vector has been damaged, and the damage information corresponding to the image corresponding to the feature vector; combine the probability value of whether the image corresponding to the feature vector has been damaged with the damage recognition loss to obtain the damage loss function of the initial convolutional neural network model.

[0126] In an exemplary embodiment, the initial convolutional neural network model is a residual network ResNet, and the feature extraction module 1150 includes an adjustment unit for adjusting the network structure of ResNet before inputting the target image set into the initial convolutional neural network model to obtain the feature vector of the target image set, so as to change the downsampling operation in ResNet from 32 times to 16 times.

[0127] As an example, the sample acquisition module 1110 is specifically used to acquire registered trademark images, counterfeit trademark images of registered trademark images, and icon images similar to registered trademark images; and use the registered trademark images, counterfeit trademark images and icon images as training image sets.

[0128] like Fig.12 As shown, Fig.12 : is a schematic diagram of an image retrieval device shown in an exemplary embodiment of the present application. The image retrieval device is based on an image retrieval model trained by the above-mentioned image retrieval model training device, and the device includes:

[0129] The image acquisition module 1210 is used to acquire the image to be identified and input the image to be identified into the image retrieval model to obtain a feature vector of the image to be identified;

[0130] The image retrieval module 1220 is used to input each retrieval image into the image retrieval model to obtain the feature vector of each retrieval image, and compare the feature vector of each retrieval image with the feature vector of the image to be identified to obtain the retrieval result of the image to be identified.

[0131] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0132] In an exemplary embodiment, an electronic device includes a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above method is implemented.

[0133] Fig.13 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment.

[0134] It should be noted that the electronic device is only an example adapted to the present application and cannot be considered to provide any limitation on the scope of use of the present application. The electronic device cannot be interpreted as requiring dependence on or having Fig.13 One or more components of an exemplary electronic device shown in FIG.

[0135] like Fig.13 As shown, in an exemplary embodiment, the electronic device includes a processing component 1301, a memory 1302, a power component 1303, a multimedia component 1304, an audio component 1305, a sensor component 1307, and a communication component 1308. Among them, the above components are not all necessary, and the electronic device can add other components or reduce certain components according to its own functional requirements, which is not limited in this embodiment.

[0136] The processing component 1301 generally controls the overall operation of the electronic device, such as operations associated with display, data communication, and log data processing. The processing component 1301 may include one or more processors 1309 to execute instructions to complete all or part of the steps of the above operations. In addition, the processing component 1301 may include one or more modules to facilitate the interaction between the processing component 1301 and other components. For example, the processing component 1301 may include a multimedia module to facilitate the interaction between the multimedia component 1304 and the processing component 1301.

[0137] The memory 1302 is configured to store various types of data to support operations on the electronic device, examples of which include instructions for any application or method for operating on the electronic device. The memory 1302 stores one or more modules, which are configured to be executed by the one or more processors 1309 to complete all or part of the steps in the method described in the above embodiment.

[0138] The power supply component 1303 provides power to various components of the electronic device. The power supply component 1303 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0139] The multimedia component 1304 includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a TP (Touch Panel) and an LCD (Liquid Crystal Display). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0140] The audio component 1305 is configured to output and / or input audio signals. For example, the audio component 1305 includes a microphone, and when the electronic device is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. In some embodiments, the audio component 1305 also includes a speaker for outputting audio signals.

[0141] The sensor component 1307 includes one or more sensors for providing status evaluation of various aspects for the electronic device. For example, the sensor component 1307 can detect the on / off state of the electronic device and can also detect the temperature change of the electronic device.

[0142] The communication component 1308 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as Wi-Fi (Wireless-Fidelity).

[0143] Understandably, Fig.13 The structure shown is for illustration only. The electronic device may include Fig.13 More or fewer components as shown, or with Fig.13 The different components are shown. Fig.13 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0144] In an exemplary embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0145] It should be noted that the computer-readable storage medium shown in the embodiment of the present application can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0146] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0147] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A method for training an image retrieval model, characterized in that: The method comprises: Acquire a training image set, wherein the training image set includes a sample image and similar samples of the sample image; Performing defect creation processing on the training image set to obtain a defect image set, and dividing the defect image set into a first defect image set and a second defect image set, performing structure destruction processing on the first defect image set to obtain a destroyed image set, wherein the images of the first defect image set are divided into n*n small blocks; Filter out text trademark images whose length 1 / n is exactly a space and graphic trademark images whose graphics are consistent before and after being damaged from the damaged image set, and use the filtered images as the second defective image set; Using the damaged image set and the second defective image set as a target image set, and inputting the target image set into an initial convolutional neural network model to obtain a feature vector of the target image set; Constructing a classifier and a damage identifier, determining a classification loss function of the initial convolutional neural network model according to the feature vector and the classification loss of the classifier, and determining a damage loss function of the initial convolutional neural network model according to the feature vector and the damage identification loss of the damage identifier; The initial convolutional neural network model is optimized according to the classification loss function and the damage loss function to obtain an image retrieval model.

2. The method according to claim 1, characterized in that: The performing defect creation processing on the training image set to obtain the defect image set comprises: Setting each mask region on the image in the training image set; The image content in each mask area is removed, or the image content in each mask area is replaced with image noise to obtain the defective image set in which part of the image information is masked.

3. The method according to claim 1, characterized in that The performing structural destruction processing on the first defective image set to obtain a damaged image set comprises: Dividing the images of the first defective image set into a plurality of small blocks; After disrupting a plurality of small blocks, a structurally destroyed image is generated to obtain the destroyed image set, and the destroyed information of the image is retained.

4. The method according to claim 1, characterized in that: Determining the classification loss function of the initial convolutional neural network model according to the feature vector and the classification loss of the classifier includes: Inputting the feature vector into the classifier to obtain a probability value that the image corresponding to the feature vector belongs to an image category; Determine the classification loss of the classifier according to the probability value that the image corresponding to the feature vector belongs to the image category and the real image category to which the image corresponding to the feature vector belongs; The probability value is combined with the classification loss of the classifier to obtain the classification loss function of the initial convolutional neural network model.

5. The method according to claim 3, characterized in that: The step of determining the damage loss function of the initial convolutional neural network model according to the feature vector and the damage recognition loss of the damage identifier includes: Inputting the feature vector into the damage identifier to obtain a probability value of whether the image corresponding to the feature vector has been damaged; Determining a damage recognition loss of the damage identifier according to a probability value of whether the image corresponding to the feature vector has been damaged and damage information corresponding to the image corresponding to the feature vector; The probability value of whether the image corresponding to the feature vector has been damaged is combined with the damage recognition loss to obtain the damage loss function of the initial convolutional neural network model.

6. The method according to any one of claims 1 to 5, characterized in that: The initial convolutional neural network model is a residual network ResNet; before inputting the target image set into the initial convolutional neural network model to obtain a feature vector of the target image set, the method further includes: The network structure of the ResNet is adjusted to modify the downsampling operation in the ResNet from 32 times to 16 times.

7. The method according to any one of claims 1 to 5, characterized in that: The sample image is a trademark image, and the step of obtaining a training image set includes: Acquire a registered trademark image, a counterfeit trademark image of the registered trademark image, and an icon image similar to the registered trademark image; The registered trademark images, counterfeit trademark images and icon images are used as the training image set.

8. An image retrieval method, characterized in that: The image retrieval method is based on an image retrieval model trained by the image retrieval model training method according to any one of claims 1 to 7, and the method comprises: Acquire an image to be identified, and input the image to be identified into the image retrieval model to obtain a feature vector of the image to be identified; Each retrieval image is input into the image retrieval model to obtain a feature vector of each retrieval image, and the feature vector of each retrieval image is compared with the feature vector of the image to be identified to obtain a retrieval result of the image to be identified.

9. An image retrieval model training device, characterized in that: include: An image set acquisition module, used to acquire a training image set, wherein the training image set includes a sample image and similar samples of the sample image; a processing module, configured to perform defect manufacturing processing on the training image set to obtain a defect image set, divide the defect image set into a first defect image set and a second defect image set, perform structural destruction processing on the first defect image set to obtain a damaged image set, wherein the images of the first defect image set are divided into n*n small blocks, and filter out text trademark images whose length 1 / n is exactly a space and graphic trademark images whose graphics are consistent before and after the destruction from the damaged image set, and use the filtered images as the second defect image set; A feature acquisition module, used to take the damaged image set and the second defective image set as a target image set, and input the target image set into an initial convolutional neural network model to obtain a feature vector of the target image set; A loss determination module, used to construct a classifier and a damage identifier, determine a classification loss function of the initial convolutional neural network model according to the feature vector and the classification loss of the classifier, and determine a damage loss function of the initial convolutional neural network model according to the feature vector and the damage identification loss of the damage identifier; A model optimization module is used to optimize the initial convolutional neural network model according to the classification loss function and the damage loss function to obtain an image retrieval model.

10. An image retrieval device, characterized in that: The image retrieval device is based on an image retrieval model trained by the image retrieval model training device according to claim 9, and the device comprises: An image acquisition module, used for acquiring an image to be identified, and inputting the image to be identified into the image retrieval model to obtain a feature vector of the image to be identified; The image retrieval module is used to input each retrieval image into the image retrieval model to obtain the feature vector of each retrieval image, and compare the feature vector of each retrieval image with the feature vector of the image to be identified to obtain the retrieval result of the image to be identified.

Images