A method and device for image feature extraction model training and image comparison

Abstract

The present invention discloses a method for training an image feature extraction model and comparing images, comprising: obtaining a sample image, performing image transformation on the sample image to obtain at least a corresponding first variation view and a second variation view; inputting the first variation view into a teacher network in a knowledge distillation network, and extracting features through the teacher network; inputting the second variation view into a student network in the knowledge distillation network, and extracting features through the student network; performing image restoration based on the features of the second variation view to obtain a restored view; determining a distillation loss based on the features of the first variation view and the features of the second variation view; determining a restoration loss based on the restored view and the sample image; and training the teacher network based on the distillation loss and the restoration loss. Accordingly, the present invention discloses an apparatus for training an image feature extraction model and comparing images.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a method and device for image similarity detection. Background Art

[0002] In this era of rapid development of mobile Internet, massive amounts of image and video data are circulated on social media and other websites every day. However, many users perform a series of modification and editing operations on images or videos, such as adding watermarks and adjusting colors, and then try to process them into new images and spread them on social media. The risk of image theft and infringement is widespread.

[0003] To mitigate these image content risks, numerous image similarity detection algorithms have been developed. Among them, algorithms based on keypoint matching, such as the Scale-Invariant Feature Transform (SIFT), are very time-consuming when working with large image libraries. Algorithms that generate hash representations of images and then compare their hash values are less effective for images with significant transformation artifacts. While deep learning-based image representation and comparison algorithms offer robustness, they still struggle to handle significant image changes and perturbations, such as mosaics, color shifts, and various types of blur.

[0004] In view of this, we hope to obtain a new image similarity detection solution to simply and efficiently solve the copyright protection problem in large-scale image scenarios. Summary of the Invention

[0005] Image similarity detection includes two parts: image feature extraction and image comparison. One of the purposes of the present invention is to provide an image feature extraction model training and image comparison method. This method performs feature extraction through a knowledge distillation network, adds an image restoration branch to it, and calculates the image restoration loss. It can effectively cope with various image transformations, weaken the impact of image changes and disturbances on the model, and improve the robustness and training efficiency of the model.

[0006] According to the above invention objectives, the present invention proposes an image feature extraction model training method, which comprises the following steps:

[0007] Acquire a sample image, and perform image transformation on the sample image to obtain at least a first changed view and a second changed view corresponding to the sample image;

[0008] Inputting the first variation view into a teacher network in a knowledge distillation network, and extracting features of the first variation view through the teacher network; inputting the second variation view into a student network in the knowledge distillation network, and extracting features of the second variation view through the student network;

[0009] performing image restoration according to the features of the second changed view to obtain a restored view;

[0010] Determine a distillation loss based on features of the first variation view and features of the second variation view; determine a restoration loss based on the restored view and the sample image;

[0011] The teacher network is trained according to the distillation loss and the restoration loss.

[0012] The present invention inputs the transformed image into a knowledge distillation network for feature extraction to obtain the distillation loss of the image, which can simply and efficiently complete the image characterization; then the extracted image features are input into the image restoration branch to calculate the image restoration loss, which can learn a feature representation that is robust to various image transformations and weaken the impact of image changes and disturbances on the model; finally, the model is trained according to the distillation loss and restoration loss, and the obtained model has high robustness and feature extraction efficiency.

[0013] Further, determining the distillation loss according to the characteristics of the first change view and the characteristics of the second change view includes:

[0014] Extracting features of the first change view through the teacher network to obtain a first feature vector, performing a centering operation on the first feature vector and then performing a normalization process;

[0015] extracting features of the second change view through the student network to obtain a second feature vector, and normalizing the second feature vector;

[0016] The distillation loss of the knowledge distillation network is calculated according to the normalized first eigenvector and the normalized second eigenvector.

[0017] Furthermore, determining the restoration loss according to the restored view and the sample image includes:

[0018] The restoration loss is calculated based on the pixel difference between the restored image and the sample image.

[0019] Furthermore, performing image transformation on the sample image includes:

[0020] Data enhancement is performed on the sample image, where the data enhancement includes image flipping, image noise addition, color change, and image blurring.

[0021] Furthermore, executing the image restoration algorithm includes:

[0022] An inverse transformation algorithm corresponding to the data enhancement is executed.

[0023] Furthermore, performing image transformation on the sample image to obtain at least a first changed view and a second changed view corresponding to the sample image includes:

[0024] Performing a global transformation on the sample image to obtain a first changed view corresponding to the sample image;

[0025] Performing a local transformation on the sample image to obtain a second changed view corresponding to the sample image.

[0026] Furthermore, the performing local transformation on the sample image includes:

[0027] A plurality of different local transformations are performed on the sample image to obtain a plurality of different second changed views.

[0028] Furthermore, training the teacher network according to the distillation loss and the restoration loss includes:

[0029] The student network parameters are updated according to the distillation loss and the restoration loss, and the parameters of the teacher network are updated according to the student network parameters.

[0030] According to the above invention objectives, the present invention further proposes an image comparison method, comprising:

[0031] Acquire detection images;

[0032] Inputting the detection image into a teacher network in a pre-trained knowledge distillation network to obtain a feature representation of the detection image, wherein the teacher network is trained using the above-mentioned image feature extraction model training method;

[0033] Determine whether the feature representation of the detection image is similar to the target image feature.

[0034] In the present invention, after the features of the detection image are extracted through the knowledge distillation network, they are compared with the pre-prepared target image features to obtain the similarity detection results between the images and determine whether the images are infringing. This greatly improves the computing speed of image similarity detection and can easily process a million-level image library.

[0035] Another object of the present invention is to provide an image feature extraction model training and image comparison device, which, by introducing an image restoration branch, can effectively cope with various image transformations, weaken the impact of image changes and disturbances on the model, and improve the robustness and training efficiency of the model.

[0036] According to the above invention objectives, the present invention proposes an image feature extraction model training device, which includes a sample processing module, a knowledge distillation module, an image restoration module and a training module:

[0037] The sample processing module acquires a sample image and performs image transformation on the sample image to obtain at least a first changed view and a second changed view corresponding to the sample image;

[0038] The knowledge distillation module inputs the first variation view into a teacher network in the knowledge distillation network, and extracts features of the first variation view through the teacher network; inputs the second variation view into a student network in the knowledge distillation network, and extracts features of the second variation view through the student network;

[0039] The image restoration module performs image restoration on the second changed view according to the characteristics of the second changed view to obtain a restored view;

[0040] The training module determines a distillation loss according to features of the first change view and features of the second change view, determines a restoration loss according to the restored view and the sample image, and trains the teacher network based on the distillation loss and the restoration loss.

[0041] The present invention obtains the first change view and the second change view through the sample processing module, obtains the image features respectively through the knowledge distillation module, and then obtains the feature representation that is robust to various image transformations through learning through the image restoration module. Finally, the model is trained in the training module; it can not only simply and efficiently complete the image representation, but also weaken the impact of image changes and disturbances on the model. The obtained model has high robustness and feature extraction efficiency.

[0042] Furthermore, the training module extracts features of the first change view through the teacher network to obtain a first feature vector, performs a centering operation on the first feature vector, and then performs normalization processing.

[0043] The features of the second change view are extracted through the student network to obtain a second feature vector, and the second feature vector is normalized.

[0044] The distillation loss of the knowledge distillation network is calculated according to the normalized first eigenvector and the normalized second eigenvector.

[0045] Furthermore, the image restoration module calculates the restoration loss according to the pixel difference between the restored image and the sample image.

[0046] Furthermore, the sample processing module performs data enhancement on the sample image, and the data enhancement includes image flipping, image noise addition, color change and image blurring.

[0047] Furthermore, the image restoration module executes an inverse transformation algorithm corresponding to the data enhancement.

[0048] Furthermore, the sample processing module performs a global transformation on the sample image to obtain a first changed view.

[0049] The sample processing module performs local transformation on the sample image to obtain a second changed view.

[0050] Furthermore, the training module updates the student network parameters according to the distillation loss and the restoration loss, and updates the parameters of the teacher network according to the student network parameters.

[0051] According to the above invention objectives, the present invention also proposes an image comparison device, including an image module, a feature extraction module and a comparison module:

[0052] The image module acquires a detection image;

[0053] The feature extraction module inputs the detection image into a teacher network in a pre-trained knowledge distillation network to obtain a feature representation of the detection image, wherein the teacher network is trained using the above-mentioned image feature extraction model training method;

[0054] The comparison module determines whether the feature representation of the detection image is similar to the feature of the target image.

[0055] In the present invention, after the features of the detection image are extracted by the feature extraction module, they are compared with the pre-prepared target image features in the comparison module to obtain the similarity detection results between the images and determine whether the images are infringing. This greatly improves the computing speed of image similarity detection and can easily process a million-level image library.

[0056] The present invention also provides a computer-readable storage medium, characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned image feature extraction model training method.

[0057] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that the processor implements the above-mentioned image feature extraction model training method when executing the program.

[0058] The image feature extraction model training and image comparison method and device described in the present invention have the following beneficial effects:

[0059] The present invention inputs the transformed image into the knowledge distillation network for feature extraction, which can simply and efficiently complete the image characterization; then the extracted image features are input into the image restoration branch, and the image restoration loss is calculated. It can learn a feature representation that is robust to various image transformations, weaken the impact of image changes and disturbances on the model, and can cope with various image transformations. The trained model has high robustness and feature extraction efficiency.

[0060] After extracting the features of the detection image through the knowledge distillation network, the present invention compares them with the pre-prepared target image features to obtain the similarity detection results between the images and determine whether the images are infringing. This greatly improves the computing speed of image similarity detection and can easily return results within seconds even for a million-level image library. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The flowchart of the image feature extraction model training method according to the present invention in one embodiment is exemplarily shown.

[0062] Figure 2 The flowchart of the image feature extraction model training method according to the present invention in a specific embodiment is exemplarily shown.

[0063] Figure 3 The flowchart of the image comparison method according to the present invention in one embodiment is shown as an example.

[0064] Figure 4 The schematic diagram of the structure of the image feature extraction model training device according to one embodiment of the present invention is exemplarily shown.

[0065] Figure 5 The figure shows a schematic structural diagram of an image comparison device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The image feature extraction model training and image comparison method and device of the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments of the specification, but this detailed description does not constitute a limitation to the present invention.

[0067] The images mentioned in the present invention include but are not limited to static pictures, dynamic pictures or single-frame images captured in video clips and other image forms. The following embodiments mainly use static pictures as examples to facilitate the explanation of the scheme, but the following embodiments are not used to limit the usage scenarios of this scheme.

[0068] In the embodiments described herein, the knowledge distillation network architecture is based on the self-supervised DINO algorithm, a commonly used knowledge distillation network architecture in computer vision. Alternatively, those skilled in the art may choose other knowledge distillation network architectures, such as DNA, AlphaNet, or TGSA, as needed.

[0069] In one embodiment of the present invention, a method for training an image feature extraction model is proposed. Figure 1The flowchart of the image feature extraction model training method according to the present invention is shown in an exemplary embodiment. The method is implemented based on a self-supervised learning algorithm. Figure 1 As shown, the specific methods include:

[0070] 100: Acquire a sample image, and perform image transformation on the sample image to obtain at least a first changed view and a second changed view corresponding to the sample image.

[0071] There are no restrictions on the source of sample images, and they can come from public datasets or pre-existing databases. There are also no restrictions on the content and format of sample images, whether they are real people, landscapes, cartoon characters, or objects; whether they are real photos or webpage screenshots, as long as the images have distinguishable features, they can be used as sample images.

[0072] Image transformations can be artificially constructed using computer programs. In some specific embodiments, image transformations can be achieved by performing data augmentation on the sample images, including image editing operations that change certain image attributes, such as image flipping, image noise addition, color change, and image blurring. Simulating various image transformations not only facilitates subsequent loss calculations, but also enables the image feature extraction model to continuously learn from these situations during training, thereby making the model more comprehensive and improving its robustness.

[0073] The first and second changed views are obtained through image transformation, which, as mentioned above, can be the result of data augmentation of the sample image. In some specific embodiments, the steps of obtaining the first and second changed views may include: performing a global transformation on the sample image to obtain the corresponding first changed view, and performing a local transformation on the sample image to obtain the corresponding second changed view. In other words, by randomly cropping all sample images into two different specific sizes and then performing random rotation, flipping, blurring, color change and other data augmentation operations, a large amount of non-repetitive first and second changed views can be obtained, fully and effectively utilizing the limited sample images. In a knowledge distillation network, a large network is usually distilled into a small network to simplify training overhead. The scale of the input samples also has a "local to global" correspondence. Therefore, it is necessary to perform local and global transformations on the sample images respectively and input them into the small and large networks respectively to ensure the rationality of the sample scale.

[0074] 110: Input the first change view into the teacher network in the knowledge distillation network, and extract features of the first change view through the teacher network; input the second change view into the student network in the knowledge distillation network, and extract features of the second change view through the student network.

[0075] Knowledge distillation networks are widely used in model compression and transfer learning. They can transfer knowledge from one network to another, either homogeneous or heterogeneous. This is accomplished by first training a teacher network and then using the teacher's output to train a student network. Knowledge distillation can be used to transform a large network into a smaller one while retaining performance close to that of the larger network. It can also be used to transfer knowledge learned from multiple networks into a single one, continuously optimizing the performance of a single network.

[0076] In the embodiment described in the present invention, the structure of the knowledge distillation network refers to the network structure of the self-supervised algorithm DINO. In the DINO network, the teacher network and the student network have exactly the same model architecture, that is, the model architecture for extracting image features, and the initial weight parameters are also exactly the same. The specific model architecture is not limited, and various models available in the prior art can be applied to this solution. You can use the emerging vision converter model (Vision Transformer, Vit) or traditional neural networks, such as Multi-Layer Perception (MLP), convolutional neural network ResNet-50, etc. The model training method described in this solution is very flexible in the choice of network structure. Of course, in some other embodiments, other knowledge distillation network structures such as DNA, AlphaNet, TGSA, etc. can also be selected.

[0077] By inputting the transformed sample image into the knowledge distillation network, the image feature extraction work can be completed better, which is not only simple to operate but also improves the calculation speed.

[0078] 120: Perform image restoration according to the features of the second changed view to obtain a restored view.

[0079] To address various image editing techniques, an image restoration branch is introduced to normalize various image transformations. In this step, the features of the extracted second-changed view are input into the image restoration branch to restore the image to the form of the sample image before the image transformation, resulting in a restored view.

[0080] In some specific embodiments, the image restoration algorithm can be implemented by executing the inverse transformation algorithm corresponding to the above-mentioned data enhancement. The inverse transformation can generally be performed directly by the decoder, or the inverse transformation program can be rewritten for various image transformations, which is not limited here.

[0081] By introducing the image restoration branch, the model can learn feature representations that are robust to various image transformations during training, mitigating the effects of image changes and perturbations. This allows the model to handle various image transformations, such as image flipping, image noise, and blurring, resulting in a highly robust and efficient feature extraction model. Furthermore, the image restoration branch described in this solution can be externalized to any available knowledge distillation network to construct an image feature extraction model, providing increased flexibility.

[0082] 130: Determine a distillation loss based on features of the first variation view and features of the second variation view; and determine a restoration loss based on the restored view and the sample image.

[0083] Distillation loss is the loss corresponding to the output of the knowledge distillation network, calculated based on the features of the first and second variation views. Distillation loss represents the difference between the extracted features of the first and second variation views, reflecting the feature extraction capability of the knowledge distillation network. A smaller distillation loss indicates a higher similarity between the features of the first and second variation views, and a stronger feature extraction capability of the knowledge distillation network, and vice versa.

[0084] Optionally, data preprocessing, such as normalization and data purification, can be performed on the extracted first and second change view features before calculating the distillation loss to simplify the loss calculation process and improve the computation speed. Loss calculation methods include, but are not limited to, mean square error (MSE) and cross entropy.

[0085] The restoration loss is the loss corresponding to the output of the image restoration branch, calculated based on the restored view and the sample image. The restoration loss represents the difference between the restored view and the sample image before the image transformation, thus reflecting the image restoration capability of the image restoration branch. The smaller the restoration loss, the closer the restored view is to the original sample image, and the better the restoration effect of the image restoration branch, and vice versa.

[0086] In some specific embodiments, the restoration loss can be calculated by calculating the pixel differences between the restored image and the sample image. Since images are composed of a series of pixels, each with corresponding attributes such as grayscale and color, calculating the difference between images can be transformed into calculating the difference between pixels. By combining the differences between all corresponding pixels, the difference between the two images to be compared can be obtained.

[0087] 140: Train the teacher network based on distillation loss and restoration loss.

[0088] Model training based on distillation loss continuously improves the feature extraction capabilities of the knowledge distillation network, obtaining increasingly powerful image features. Model training based on restoration loss improves the image restoration capabilities of the image restoration branch and enables the model to learn feature representations that are robust to various image transformations. This mitigates the impact of image changes and perturbations on the model, allowing it to cope with various image transformations and resulting in a highly robust model.

[0089] It should be noted that in the knowledge distillation network, the performance of the teacher network is the overall training goal.

[0090] In some specific embodiments, the teacher network is trained based on the distillation loss and the restoration loss. Specifically, the student network parameters can be updated based on the distillation loss and the restoration loss, and the teacher network parameters can be updated based on the student network parameters. Therefore, the teacher network does not directly participate in training. Instead, after each round of training, the model backpropagates the losses to the student network. Based on the parameter updates of the student network, the teacher network can also complete the parameter updates.

[0091] In some more specific embodiments, the exponential moving average (EMA) of the student network parameters is used to update the parameters of the teacher network.

[0092] In some specific embodiments, the step of obtaining the first and second changed views may include performing a global transformation on the sample images to obtain the corresponding first changed view, and performing a local transformation on the sample images to obtain the corresponding second changed view. In other words, by randomly cropping all sample images into two different specific sizes and then performing random data augmentation operations such as rotation, flipping, blurring, and color changes, a large amount of non-repetitive first and second changed views can be obtained, fully and effectively utilizing the limited sample images.

[0093] In some more specific embodiments, the local transformation of the sample image can be performed by performing multiple different local transformations on the sample image to obtain multiple different second modified views. In other words, the number of first modified views and second modified views does not necessarily have to be the same, and relevant technicians in this field can specify it based on specific model characteristics and training requirements.

[0094] Optionally, in some specific embodiments, the calculation of distillation loss includes:

[0095] Extract the features of the first change view through the teacher network to obtain the first eigenvector, perform a centering operation on the first eigenvector and then perform normalization processing;

[0096] Extracting features of the second change view through the student network to obtain a second eigenvector, and normalizing the second eigenvector;

[0097] The distillation loss of the knowledge distillation network is calculated based on the normalized first eigenvector and the normalized second eigenvector.

[0098] Since the number of the first change view and the second change view are inconsistent in some cases, there are differences in the dimensions of the feature vectors extracted by the corresponding teacher network and student network. Therefore, before calculating the distillation loss, a normalization preprocessing process can be introduced to the obtained feature vector, which can usually be achieved using a function. In some more specific embodiments, an exponential normalization function (Softmax function) is used to normalize the first eigenvector and the second eigenvector. The Softmax function can map the output value range of each output node to [0,1], and constrain the sum of the output values of each output node to be 1. In the solution described in the present invention, after normalization, the first eigenvector and the second eigenvector can be used to calculate the distillation loss in the same format, which can simplify the loss calculation process and improve the calculation rate.

[0099] Among them, centering the first eigenvector can obtain the central feature of the first eigenvector by calculating the average value of the output results of the teacher network. This can avoid the dominance of single-dimensional data. Combined with normalization, it can fully prevent the model from collapsing, improve the stability of the model, and make the feature representation of the image more unified, making the feature extraction results more accurate.

[0100] Figure 2 The flowchart of the image feature extraction model training method according to the present invention in a specific embodiment is exemplarily shown.

[0101] like Figure 2 As shown, in this specific embodiment, the sample image I is first i Perform image transformation to obtain a corresponding first changed view and a second changed view.

[0102] Image transformation can be achieved by performing data augmentation on sample images. This includes image editing operations that change certain image attributes, such as image flipping, image noise addition, color change, and image blurring. This not only facilitates subsequent loss calculations but also improves the robustness of the model. A global transformation of the sample image yields the corresponding first altered view, while a local transformation of the sample image yields the corresponding second altered view. In other words, all sample images are randomly cropped into two different specific sizes, and then subjected to random data augmentation operations such as rotation, flipping, blurring, and color change.

[0103] More specifically, the local transformation of the sample image can be performed multiple times by performing different local transformations on the sample image to obtain multiple different second modified views. In other words, the number of first modified views and second modified views does not necessarily have to be the same, and relevant technicians in this field can specify it based on specific model characteristics and training requirements.

[0104] Then the first change view is input into the teacher network in the knowledge distillation network, and the features of the first change view are extracted through the teacher network; the second change view is input into the student network E in the knowledge distillation network, and the features of the second change view are extracted through the student network.

[0105] In the embodiment described in the present invention, the structure of the knowledge distillation network refers to the network structure of the self-supervised algorithm DINO. In the DINO network, the teacher network and the student network have exactly the same model architecture, that is, the model architecture for extracting image features, and the initial weight parameters are also exactly the same. The specific model architecture is not limited, and various models available in the prior art can be applied to this solution, which is very flexible. In this embodiment, the knowledge distillation network adopts a self-distillation algorithm, and a multi-layer perceptron (MLP) can be selected for image feature extraction. As a relatively basic neural network model, MLP takes as input the transformed view of the sample image and outputs the feature vectors corresponding to each view. It can better complete the image feature extraction work, which is not only simple to operate, but also improves the calculation speed.

[0106] Then, image restoration is performed according to the features of the second changed view to obtain a restored view.

[0107] While the student network extracts features from the second variation view, image restoration is also required. In some specific embodiments, this image restoration algorithm can be implemented by executing the inverse transformation algorithm corresponding to the aforementioned data augmentation. In this embodiment, image restoration is performed directly on decoder D, which has an embedded inverse transformation algorithm. Of course, the inverse transformation program can also be rewritten for various image transformations. This step yields several restored views similar to the initial sample image.

[0108] By introducing an image restoration branch, we can learn feature representations that are robust to various image transformations, mitigate the impact of image changes and perturbations on the model, and adapt to various image transformations, such as image flipping, image noise, and blurring. This ensures that the trained model has high robustness and feature extraction efficiency. Furthermore, the image restoration branch described in this solution can be externalized to any available existing model, providing a certain degree of flexibility.

[0109] Then, the distillation loss can be determined according to the features of the first change view and the features of the second change view; and the restoration loss can be determined according to the restored view and the sample image.

[0110] Optionally, data preprocessing, such as normalization and data purification, can be performed on the extracted first and second change view features before calculating the distillation loss to simplify the loss calculation process and improve the computation speed. Loss calculation methods include, but are not limited to, mean square error (MSE) and cross entropy.

[0111] In this example, the calculation for distillation loss includes:

[0112] Extract the features of the first change view through the teacher network to obtain the first eigenvector, perform a centering operation on the first eigenvector and then perform normalization processing;

[0113] Extracting features of the second change view through the student network to obtain a second eigenvector, and normalizing the second eigenvector;

[0114] The distillation loss of the knowledge distillation network is calculated based on the normalized first eigenvector p1 and the normalized second eigenvector p2.

[0115] Since the dimensions of the first eigenvector and the second eigenvector are different in some cases, a normalization preprocessing process can be introduced to the obtained eigenvector before calculating the distillation loss, which can usually be achieved using a function. In this embodiment, the Softmax function is used to perform exponential normalization on the first eigenvector and the second eigenvector. The Softmax function can map the output value range of each output node to [0,1] and constrain the sum of the output values of each output node to be 1. After normalization, the first eigenvector and the second eigenvector can be used in the same format to calculate the distillation loss, which can simplify the loss calculation process and improve the calculation rate.

[0116] Among them, centering the first eigenvector can obtain the central feature of the first eigenvector by calculating the average value of the output results of the teacher network. This can avoid the excessive prominence of a certain dimension feature in the eigenvector, better represent the characteristics of the image, and make the feature extraction results more accurate.

[0117] The distillation loss L1 of the knowledge distillation network calculated based on the normalized first eigenvector p1 and the normalized second eigenvector p2 is as follows:

[0118] L1=-p2log(p1)

[0119] In this embodiment, the restoration loss can be calculated by calculating the pixel differences between the restored image and the sample image. Since images are composed of a series of pixels, each with corresponding attributes such as grayscale and color, calculating the difference between images can be transformed into calculating the difference between pixels. By combining the differences between all corresponding pixels, the difference between the two images to be compared can be obtained.

[0120] According to the restored view and sample image I i The determined restoration loss L2 is as follows:

[0121] L2=‖D(E(I i ))-I i ‖2

[0122] Among them, E(I i ) represents the sample image I extracted by the student network i The second eigenvector of i )) represents the sample image I i The corresponding restored view; i is the serial number of the sample image.

[0123] Finally, the teacher network is trained according to the distillation loss L1 and the restoration loss L2 to obtain the desired image feature extraction model.

[0124] It should be noted that in the knowledge distillation network, the performance of the teacher network is the overall training goal.

[0125] In this embodiment, the teacher network is trained based on the distillation loss and restoration loss. Specifically, the student network parameters are updated based on the distillation loss and restoration loss, and the teacher network parameters are updated based on the student network parameters. Therefore, the teacher network does not directly participate in training. Instead, after each round of training, the model backpropagates the losses to the student network. Based on the parameter updates of the student network, the teacher network also completes the parameter updates.

[0126] More specifically, the exponential moving average (EMA) of the student network parameters is used to update the parameters of the teacher network.

[0127] In one embodiment of the present invention, an image comparison method is also proposed, such as Figure 3 As shown:

[0128] 300: Acquire a detection image.

[0129] When acquiring test images, they are typically organized into a database for batch similarity checking. Alternatively, images can be downloaded or captured from the internet for individual similarity checks. There are no restrictions on the source or quantity of images. The content and format of the test images are equally open, whether they are real people, landscapes, cartoon characters, or objects; whether they are real photos or webpage screenshots; as long as the images share extractable features, image comparison can be performed.

[0130] 310: Input the detection image into the teacher network in the pre-trained knowledge distillation network to obtain the feature representation of the detection image, wherein the teacher network is trained using the above-mentioned image feature extraction model training method.

[0131] Since the above-mentioned image feature extraction model added an image restoration branch during training, the trained teacher network learned the feature representation of various image transformations, has good robustness, can flexibly respond to various image transformations such as image flipping, image noise, blurring, etc., and can accurately and quickly extract effective features.

[0132] It should be noted that in some embodiments, the target image can first be extracted from a pre-trained teacher network in a knowledge distillation network, and then the features can be stored in a feature library. Of course, feature extraction can also be performed simultaneously on both the detection image and the target image. Those skilled in the art can freely choose according to specific detection requirements.

[0133] 320: Determine whether the feature representation of the detection image is similar to the target image feature.

[0134] Alternatively, determining whether the feature representation of the test image is similar to that of the target image can be accomplished by comparing the cosine distance. The comparison result is expressed as a similarity. If the similarity exceeds a threshold, the test image is considered similar to the target image; vice versa. Alternatively, other similarity comparison methods such as Euclidean distance, Manhattan distance, and Chebyshev distance can be used.

[0135] In this era of rapid development of mobile Internet, massive amounts of image and video data are disseminated on social media and other websites every day. However, many users attempt to process images or videos into new images and disseminate them on social media after performing a series of modification and editing operations such as adding watermarks and adjusting colors. This poses a widespread risk of image theft and infringement, so it is necessary to perform similarity detection on images to determine whether there is image piracy or infringement. Compared with other existing similarity detection methods, the image comparison method described in the embodiment of the present invention can not only efficiently process large-scale image libraries, but also effectively cope with images with strong changes. It has high robustness and can be used to simply and efficiently solve copyright protection problems and other image similarity detection problems in large-scale image scenarios.

[0136] In another embodiment of the present invention, an image feature extraction model training device is proposed. Figure 4 The following is a schematic diagram showing the structure of the image feature extraction model training device according to one embodiment of the present invention. Figure 4 As shown, it includes a sample processing module, a knowledge distillation module, an image restoration module and a training module:

[0137] The sample processing module 40 acquires a sample image and performs image transformation on the sample image to obtain at least a first changed view and a second changed view corresponding to the sample image.

[0138] There are no restrictions on the source of sample images, and they can come from public datasets or pre-existing databases. There are also no restrictions on the content and format of sample images, whether they are real people, landscapes, cartoon characters, or objects; whether they are real photos or webpage screenshots, as long as the images have distinguishable features, they can be used as sample images.

[0139] In the sample processing module, image transformations can be artificially constructed using computer programs. In some specific embodiments, image transformations can be achieved by performing data augmentation on the sample images, including image editing operations that change certain image attributes, such as image flipping, image noise addition, color change, and image blurring. Simulating various image transformations not only facilitates subsequent loss calculations, but also enables the image feature extraction model to continuously learn from these situations during training, thereby making the model more comprehensive and improving its robustness.

[0140] The first and second changed views are obtained through image transformation and are the result of data augmentation of the sample images. In some specific embodiments, the steps of obtaining the first and second changed views may include: performing a global transformation on the sample image to obtain the corresponding first changed view, and performing a local transformation on the sample image to obtain the corresponding second changed view. In other words, by randomly cropping all sample images into two different specific sizes and then performing random data augmentation operations such as rotation, flipping, blurring, and color changes, a large amount of non-repetitive first and second changed views can be obtained, fully and effectively utilizing the limited sample images.

[0141] The knowledge distillation module 42 inputs the first change view into the teacher network in the knowledge distillation network, and extracts the features of the first change view through the teacher network; inputs the second change view into the student network in the knowledge distillation network, and extracts the features of the second change view through the student network.

[0142] In the embodiment described in the present invention, the network structure of the knowledge distillation module refers to the network structure of the self-supervised algorithm DINO. In the DINO network, the teacher network and the student network have exactly the same model architecture, that is, the model architecture for extracting image features, and the initial weight parameters are also exactly the same. The specific model architecture is not limited, and various models available in the prior art can be applied to this solution. You can use the emerging vision converter model (Vision Transformer, Vit) or traditional neural networks, such as Multi-Layer Perception (MLP), convolutional neural network ResNet-50, etc. The model training method described in this solution is very flexible in the choice of network structure. Of course, in some other embodiments, other knowledge distillation network structures such as DNA, AlphaNet, TGSA, etc. can also be selected.

[0143] By inputting the transformed sample image into the knowledge distillation network, the image feature extraction work can be completed better, which is not only simple to operate but also improves the calculation speed.

[0144] The image restoration module 44 performs image restoration on the second changed view according to the characteristics of the second changed view to obtain a restored view.

[0145] To address various image editing techniques, an image restoration module is introduced to normalize various image transformations. In this module, the features of the extracted second-changed view are input into the image restoration branch, restoring the image to the form of the sample image before the image transformation, resulting in a restored view.

[0146] In some specific embodiments, the image restoration algorithm can be implemented by executing the inverse transformation algorithm corresponding to the above-mentioned data enhancement. The inverse transformation can generally be performed directly by the decoder, or the inverse transformation program can be rewritten for various image transformations, which is not limited here.

[0147] By introducing an image restoration module, we can learn feature representations that are robust to various image transformations, mitigate the impact of image changes and perturbations on the model, and adapt to various image transformations, such as image flipping, image noise, and blurring. This ensures that the trained model has high robustness and feature extraction efficiency. Furthermore, the image restoration branch described in this solution can be externalized to any available existing model, providing a certain degree of flexibility.

[0148] The training module 46 determines a distillation loss according to features of the first change view and features of the second change view, determines a restoration loss according to the restored view and the sample image, and trains the teacher network based on the distillation loss and the restoration loss.

[0149] The distillation loss is the loss corresponding to the output result of the knowledge distillation network. It is calculated based on the features of the first change view and the second change view, reflecting the feature extraction ability of the knowledge distillation network. The smaller the distillation loss, the higher the feature similarity between the first change view and the second change view, and the stronger the feature extraction ability of the knowledge distillation network, and vice versa.

[0150] Optionally, data preprocessing, such as normalization and data purification, can be performed on the extracted first and second change view features before calculating the distillation loss to simplify the loss calculation process and improve the computation speed. Loss calculation methods include, but are not limited to, mean square error (MSE) and cross entropy.

[0151] The restoration loss is the loss corresponding to the output result of the image restoration branch. It is calculated based on the restored view and sample image, and can reflect the image restoration ability of the image restoration branch. The smaller the restoration loss, the closer the restored view is to the original sample image, and the better the restoration effect of the image restoration branch, and vice versa.

[0152] In some specific embodiments, the restoration loss can be calculated by calculating the pixel differences between the restored image and the sample image. Since images are composed of a series of pixels, each with corresponding attributes such as grayscale and color, calculating the difference between images can be transformed into calculating the difference between pixels. By combining the differences between all corresponding pixels, the difference between the two images to be compared can be obtained.

[0153] Model training based on distillation loss continuously improves the feature extraction capabilities of the knowledge distillation network, obtaining increasingly powerful image features. Model training based on restoration loss improves the image restoration capabilities of the image restoration branch and enables the model to learn feature representations that are robust to various image transformations. This mitigates the impact of image changes and perturbations on the model, allowing it to cope with various image transformations and resulting in a highly robust model.

[0154] It should be noted that in the knowledge distillation network, the performance of the teacher network is the overall training goal.

[0155] In some specific embodiments, the teacher network is trained based on the distillation loss and the restoration loss. Specifically, the student network parameters can be updated based on the distillation loss and the restoration loss, and the teacher network parameters can be updated based on the student network parameters. Therefore, the teacher network does not directly participate in training. Instead, after each round of training, the model backpropagates the losses to the student network. Based on the parameter updates of the student network, the teacher network can also complete the parameter updates.

[0156] In some more specific embodiments, the exponential moving average (EMA) of the student network parameters is used to update the parameters of the teacher network.

[0157] In some specific embodiments, the step of obtaining the first and second changed views in the sample processing module may include performing a global transformation on the sample image to obtain the corresponding first changed view, and performing a local transformation on the sample image to obtain the corresponding second changed view. In other words, by randomly cropping all sample images to two different specific sizes and then performing random data augmentation operations such as rotation, flipping, blurring, and color changes, a large amount of non-repetitive first and second changed views can be obtained, fully and effectively utilizing the limited sample images.

[0158] In some more specific embodiments, the local transformation of the sample image can be performed by performing multiple different local transformations on the sample image to obtain multiple different second modified views. In other words, the number of first modified views and second modified views does not necessarily have to be the same, and relevant technicians in this field can specify it based on specific model characteristics and training requirements.

[0159] Optionally, in some specific embodiments, the calculation of the distillation loss in the training module includes:

[0160] Extract the features of the first change view through the teacher network to obtain the first eigenvector, perform a centering operation on the first eigenvector and then perform normalization processing;

[0161] Extracting features of the second change view through the student network to obtain a second eigenvector, and normalizing the second eigenvector;

[0162] The distillation loss of the knowledge distillation network is calculated based on the normalized first eigenvector and the normalized second eigenvector.

[0163] Since the number of the first change view and the second change view are inconsistent in some cases, there are differences in the dimensions of the feature vectors extracted by the corresponding teacher network and student network. Therefore, before calculating the distillation loss, a normalization preprocessing process can be introduced to the obtained feature vector, which can usually be achieved using a function. In some more specific embodiments, an exponential normalization function (Softmax function) is used to normalize the first eigenvector and the second eigenvector. The Softmax function can map the output value range of each output node to [0,1], and constrain the sum of the output values of each output node to be 1. In the solution described in the present invention, after normalization, the first eigenvector and the second eigenvector can be used to calculate the distillation loss in the same format, which can simplify the loss calculation process and improve the calculation rate.

[0164] Among them, centering the first eigenvector can be performed by calculating the average value of the output results of the teacher network to obtain the central feature of the first eigenvector. This can avoid the excessive prominence of a certain dimension feature in the eigenvector, better represent the characteristics of the image, and make the feature extraction results more accurate.

[0165] In another embodiment of the present invention, an image comparison device is provided. Figure 5 The following is a schematic diagram showing the structure of the image comparison device according to the present invention in one embodiment. Figure 5 As shown, it includes image module, feature extraction module and comparison module:

[0166] The image module 50 acquires a detection image.

[0167] In the image module, images to be tested are typically organized into a database for batch similarity checking. Alternatively, images can be downloaded or captured from the internet for individual similarity checks. There are no restrictions on the source or quantity of images. The content and format of the images tested are also unrestricted, whether they are real people, landscapes, cartoon characters, or objects; whether they are real photos or webpage screenshots; as long as the images share extractable features, image comparison can be performed.

[0168] The feature extraction module 52 inputs the detection image into a teacher network in a pre-trained knowledge distillation network to obtain a feature representation of the detection image, wherein the teacher network is trained using the above-mentioned image feature extraction model training method.

[0169] In the feature extraction module, since the above-mentioned image feature extraction model added an image restoration branch during training, the trained teacher network learned the feature representation of various image transformations, has good robustness, and can flexibly respond to various image transformations such as image flipping, image noise, and blurring, and accurately and quickly extract effective features.

[0170] The comparison module 54 determines whether the feature representation of the detection image is similar to the feature representation of the target image.

[0171] In some embodiments, the target image can first be extracted from a pre-trained teacher network in a knowledge distillation network, and then the features can be stored in a feature library. Of course, feature extraction can also be performed simultaneously on both the test image and the target image. Those skilled in the art can freely choose according to specific detection requirements.

[0172] Alternatively, determining whether the feature representation of the test image is similar to that of the target image can be accomplished by comparing the cosine distance. The comparison result is expressed as a similarity. If the similarity exceeds a threshold, the test image is considered similar to the target image; vice versa. Alternatively, other similarity comparison methods such as Euclidean distance, Manhattan distance, and Chebyshev distance can be used.

[0173] The solution described in this invention provides a method and apparatus for training an image feature extraction model and image comparison. This method uses a transformed image as input to a knowledge distillation network for feature extraction, obtaining the image's distillation loss. This allows for simple and efficient image representation. The extracted image features are then input to an image restoration branch, where the image restoration loss is calculated. This method learns a feature representation that is robust to various image transformations, mitigating the effects of image changes and perturbations on the model. Finally, the model is trained based on the distillation and restoration losses, resulting in a model with high robustness and feature extraction efficiency. This solution can be applied to fields such as image theft and infringement detection.

[0174] One embodiment of the present invention provides a computer-readable storage medium, characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned image feature extraction model training method.

[0175] An embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned image feature extraction model training method when executing the program.

[0176] It should be noted that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many similar variations are possible. All variations directly derived from or associating with the present invention by those skilled in the art are intended to fall within the scope of protection of the present invention.

Claims

1. A method for training an image feature extraction model, comprising: Acquire a sample image, and perform image transformation on the sample image to obtain at least a first changed view and a second changed view corresponding to the sample image; Inputting the first variation view into a teacher network in a knowledge distillation network, and extracting features of the first variation view through the teacher network; inputting the second variation view into a student network in the knowledge distillation network, and extracting features of the second variation view through the student network; performing image restoration according to the features of the second changed view to obtain a restored view; determining a distillation loss based on characteristics of the first variation view and characteristics of the second variation view; determining a restoration loss according to the restored view and the sample image; The teacher network is trained according to the distillation loss and the restoration loss.

2. The image feature extraction model training method according to claim 1, wherein determining the distillation loss based on the features of the first variation view and the features of the second variation view comprises: Extracting features of the first change view through the teacher network to obtain a first feature vector, performing a centering operation on the first feature vector and then performing a normalization process; extracting features of the second change view through the student network to obtain a second feature vector, and normalizing the second feature vector; The distillation loss of the knowledge distillation network is calculated according to the normalized first eigenvector and the normalized second eigenvector.

3. The image feature extraction model training method according to claim 1, wherein determining the restoration loss based on the restored view and the sample image comprises: The restoration loss is calculated according to the pixel difference between the restored image and the sample image.

4. The image feature extraction model training method according to claim 1, wherein performing image transformation on the sample image comprises: Data enhancement is performed on the sample image, where the data enhancement includes image flipping, image noise addition, color change, and image blurring.

5. The image feature extraction model training method according to claim 4, wherein executing the image restoration algorithm comprises: An inverse transformation algorithm corresponding to the data enhancement is executed.

6. The image feature extraction model training method according to claim 1, wherein performing image transformation on the sample image to obtain at least a first changed view and a second changed view corresponding to the sample image comprises: Performing a global transformation on the sample image to obtain a first changed view corresponding to the sample image; Performing a local transformation on the sample image to obtain a second changed view corresponding to the sample image.

7. The image feature extraction model training method according to claim 6, wherein the performing local transformation on the sample image comprises: A plurality of different local transformations are performed on the sample image to obtain a plurality of different second changed views.

8. The image feature extraction model training method according to claim 1, wherein training the teacher network according to the distillation loss and the restoration loss comprises: The student network parameters are updated according to the distillation loss and the restoration loss, and the parameters of the teacher network are updated according to the student network parameters.

9. An image comparison method, comprising: Acquire detection images; Inputting the detection image into a teacher network in a pre-trained knowledge distillation network to obtain a feature representation of the detection image, wherein the teacher network is trained using the method according to any one of claims 1 to 8; Determine whether the feature representation of the detection image is similar to the target image feature.

10. An image feature extraction model training device, comprising a sample processing module, a knowledge distillation module, an image restoration module, and a training module: The sample processing module acquires a sample image and performs image transformation on the sample image to obtain at least a first changed view and a second changed view corresponding to the sample image; The knowledge distillation module inputs the first variation view into a teacher network in the knowledge distillation network, and extracts features of the first variation view through the teacher network; inputs the second variation view into a student network in the knowledge distillation network, and extracts features of the second variation view through the student network; The image restoration module performs image restoration on the second changed view according to the characteristics of the second changed view to obtain a restored view; The training module determines a distillation loss according to features of the first change view and features of the second change view, determines a restoration loss according to the restored view and the sample image, and trains the teacher network based on the distillation loss and the restoration loss.

11. The image feature extraction model training device according to claim 10, The training module extracts features of the first change view through the teacher network to obtain a first feature vector, performs a centering operation on the first feature vector, and then performs a normalization process; extracting features of the second change view through the student network to obtain a second feature vector, and normalizing the second feature vector; The distillation loss of the knowledge distillation network is calculated according to the normalized first eigenvector and the normalized second eigenvector.

12. The image feature extraction model training device according to claim 10, The image restoration module calculates the restoration loss according to the pixel difference between the restored image and the sample image.

13. The image feature extraction model training device according to claim 10, The sample processing module performs data enhancement on the sample image, where the data enhancement includes image flipping, image noise addition, color change, and image blurring.

14. The image feature extraction model training device according to claim 13, The image restoration module executes an inverse transformation algorithm corresponding to the data enhancement.

15. The image feature extraction model training device according to claim 10, The sample processing module performs a global transformation on the sample image to obtain a first changed view; The sample processing module performs local transformation on the sample image to obtain a second changed view.

16. The image feature extraction model training device according to claim 15, The training module updates the student network parameters according to the distillation loss and the restoration loss, and updates the parameters of the teacher network according to the student network parameters.

17. An image comparison device, comprising an image module, a feature extraction module, and a comparison module: The image module acquires a detection image; The feature extraction module inputs the detection image into the teacher network in the pre-trained knowledge distillation network to obtain the feature representation of the detection image, wherein, The teacher network is trained using the method according to any one of claims 1 to 8; The comparison module determines whether the feature representation of the detection image is similar to the feature of the target image.

18. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images