Image similarity determination method and training method and device for model thereof

Abstract

The application provides an image similarity determination method and a training method and device of a model thereof, generates training data based on a preset first sample set; inputs the training data into an initial model, extracts first image features of images in a positive sample combination and second image features of images in a negative sample combination through the initial model; determines a loss value of the initial model based on the first image features and the second image features; updates model parameters of the initial model based on the loss value; continues to train the model through the training data until the loss value converges, and determines the initial model after the loss value converges as an image similarity determination model. The above method determines the positive sample combination and the negative sample combination based on images generated by sample image transformation, trains the image similarity determination model through a self-supervised manner, does not need to obtain a large amount of labeled data to train the image similarity determination model, reduces data acquisition cost, and improves training efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of memory detection, in particular to an image similarity determination method and a training method and device of a model thereof. BACKGROUND

[0002] In the related art, the similarity between images is usually calculated based on the original pixel information of the images, or the distance between features is calculated after the characteristics of the images are extracted by a manually designed operator, so as to determine the similarity between the images according to the calculation result. However, the calculation based on the original pixel information of the images has a large noise in the similarity result, and the features extracted by the manual operator have certain limitations in the scope of application, resulting in a low rationality of the similarity result. In addition, a similarity determination model can be established based on deep learning theory to extract features of images, and then the similarity between images is determined based on the distance between the features, but a large amount of labeled image data is required to train the model, which has a high data acquisition cost and low efficiency. SUMMARY

[0003] Therefore, the present application aims to provide an image similarity determination method and a training method and device of a model thereof, so as to improve the monitoring efficiency of memory usage and reduce the labor cost.

[0004] In a first aspect, the present application provides a training method of an image similarity determination model, comprising: generating training data based on a preset first sample set; the first sample set comprises a plurality of sample images; the training data comprises a positive sample combination and a negative sample combination; the positive sample combination comprises two images, both of which are obtained by transforming a first sample image; the negative sample combination comprises two images, which are obtained by transforming the first sample image and a second sample image, respectively; inputting the training data into an initial model, extracting first image features of the images in the positive sample combination and second image features of the images in the negative sample combination by the initial model; determining a loss value of the initial model based on the first image features and the second image features; updating model parameters of the initial model based on the loss value; continuing to perform the step of generating training data based on the preset sample set until the loss value converges, and determining the initial model after the loss value converges as the image similarity determination model.

[0005] The step of generating training data based on the preset first sample set comprises: determining a first sample image and a second sample image from the plurality of sample images; performing image processing on the first image sample to obtain a plurality of first transformed images; performing image processing on the second image sample to obtain a plurality of second transformed images; determining a first image and a second image in the first transformed images as the positive sample combination; and determining a third image in the first transformed images and a fourth image in the second transformed images as the negative sample combination.

[0006] The first image features include first features of the first image and second features of the second image in the positive sample combination, and the second image features include third features of the third image and fourth features of the fourth image in the negative sample combination; the step of determining the loss value of the initial model based on the first image features and the second image features includes: calculating the loss value of the initial model by a loss function of a contrast learning algorithm based on the first features of the first image and the second features of the second image in the positive sample combination and the third features of the third image and the fourth features of the fourth image in the negative sample combination.

[0007] In a second aspect, an embodiment of the present application provides a method for determining image similarity, including: obtaining a first target image and a second target image; inputting the first target image and the second target image into a pre-trained image similarity determination model, and outputting first target features of the first target image and second target features of the second target image through the image similarity determination model; the image similarity model is trained by the training method of the image similarity determination model; and determining similarity of the first target image and the second target image based on the first target features and the second target features.

[0008] The step of determining the similarity of the first target image and the second target image based on the first target features and the second target features includes: calculating Euclidean distance and cosine distance between the first target features and the second target features; and determining the similarity of the first target image and the second target image based on the Euclidean distance and the cosine distance between the first target features and the second target features.

[0009] The method further includes: determining whether the similarity of the first target image and the second target image is greater than or equal to a target threshold value; if yes, generating a plurality of image combinations based on the first target image and the second target image; each image combination includes a first sub-image of the first target image and a second sub-image of the second target image corresponding to the first sub-image; position information of the image combination corresponds to position information of the first sub-image in the first target image; for each image combination, inputting the first sub-image and the second sub-image into a pre-trained feature extraction network, extracting multi-dimensional features of the first sub-image and the second sub-image in the image combination through the feature extraction network; inputting the multi-dimensional features of the first sub-image and the second sub-image into a pre-trained similarity measurement network, processing the multi-dimensional features through the similarity measurement network, and determining a perceptual similarity of the first sub-image and the second sub-image; determining a similarity parameter of the first target image and the second target image based on the position information of the image combination and the perceptual similarity corresponding to the image combination; and the similarity parameter is used to represent a local similarity of the first target image and the second target image.

[0010] The step of generating the plurality of image combinations based on the first target image and the second target image comprises: dividing the first target image into a plurality of first sub-images, and dividing the second target image into a plurality of second sub-images corresponding to the first sub-images; determining the first sub-image and the second sub-image corresponding to the first sub-image as an image combination, and determining position information of the first sub-image in the first target image as position information of the image combination.

[0011] The feature extraction network comprises an AlexNet network; the AlexNet network comprises a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence; the step of extracting the multi-dimensional features of the first sub-image and the second sub-image in the image combination by the feature extraction network comprises: performing feature extraction processing on the first sub-image and the second sub-image in the image combination by the AlexNet network to obtain a plurality of first sub-features of the first sub-image and a plurality of second sub-features of the second sub-image, respectively; the plurality of first sub-features are output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively; the second sub-features are output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively; the plurality of first sub-features of the first sub-image are normalized by the second norm to obtain the multi-dimensional features corresponding to the first sub-image; and the plurality of second sub-features of the second sub-image are normalized by the second norm to obtain the multi-dimensional features corresponding to the second sub-image.

[0012] The similarity measurement network is trained in the following manner: generating training data based on a preset second sample set; the training data comprises an original sample image, and a first degraded image and a second degraded image corresponding to the original sample image; inputting the training data into the pre-trained feature extraction network to extract multi-dimensional features in the original sample image, the first degraded image, and the second degraded image by the feature extraction network; inputting the multi-dimensional features in the original sample image, the first degraded image, and the second degraded image into the initial network to determine a first perceptual similarity between the original sample image and the first degraded image, and a second perceptual similarity between the original sample image and the second degraded image by the initial network; processing the first perceptual similarity and the second perceptual similarity by a preset judge to obtain a perceptual similarity accuracy parameter; the judge is established based on a multi-layer perceptron network; calculating a loss value of the training process based on the first perceptual similarity, the second perceptual similarity, and the perceptual similarity accuracy parameter by a cross-entropy loss function; updating model parameters of the initial network and the judge based on the loss value; continuing to perform the step of generating the training data based on the preset sample set until the loss value converges, and determining the initial network after the loss value converges as the similarity measurement network.

[0013] The similarity measurement network comprises a plurality of sub-feature networks and an output network; the plurality of sub-feature networks are connected with the output network respectively; the sub-feature network comprises a convolution layer and an average pooling layer; the convolution layer of each sub-feature network corresponds to a set of convolution parameters; the step of determining the perceptual similarity of the first sub-image and the second sub-image by processing the multi-dimensional features through the similarity measurement network comprises: calculating the feature distance between the multi-dimensional features of the first sub-image and the multi-dimensional features of the second sub-image; extracting a plurality of distance features of the feature distance through the plurality of sub-feature networks respectively; the dimensions of the distance features correspond to the convolution parameters of the sub-feature network for extracting the distance features; and calculating the perceptual similarity of the first sub-image and the second sub-image based on the plurality of distance features through the output network.

[0014] In a third aspect, an embodiment of the present application provides a training device of an image similarity determination model, comprising: a training data generation module configured to generate training data based on a preset first sample set; the first sample set comprises a plurality of sample images; the training data comprises a positive sample combination and a negative sample combination; the positive sample combination comprises two images, both of which are obtained by transformation of a first sample image; the negative sample combination comprises two images, both of which are obtained by transformation of a first sample image and a second sample image; a feature extraction module configured to input the training data into an initial model, extract first image features of the images in the positive sample combination and second image features of the images in the negative sample combination through the initial model; a first loss value determination module configured to determine a loss value of the initial model based on the first image features and the second image features; a parameter update module configured to update model parameters of the initial model based on the loss value; continue to perform the step of generating training data based on the preset sample set until the loss value converges, and determine the initial model after the loss value converges as the image similarity determination model.

[0015] In a fourth aspect, an embodiment of the present application provides an image similarity determination device, comprising: a target image acquisition module configured to acquire a first target image and a second target image; a feature output module configured to input the first target image and the second target image into a pre-trained image similarity determination model, and output first target features of the first target image and second target features of the second target image through the image similarity determination model; the image similarity model is trained by the training method of the image similarity determination model; a similarity determination module configured to determine the similarity of the first target image and the second target image based on the first target features and the second target features.

[0016] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the training method of the image similarity determination model or the image similarity determination method.

[0017] In a sixth aspect, an embodiment of the present application provides a machine readable storage medium storing machine executable instructions, which, when invoked and executed by a processor, cause the processor to implement the training method of the image similarity determination model or the image similarity determination method.

[0018] The embodiments of the present application bring the following beneficial effects:

[0019] The image similarity determination method, the training method of the image similarity determination model and the device, generate training data based on the first sample set; input the training data into the initial model, extract the first image features of the images in the positive sample combination and the second image features of the images in the negative sample combination through the initial model; determine the loss value of the initial model based on the first image features and the second image features; update the model parameters of the initial model based on the loss value; continue to train the model through the training data until the loss value converges, and determine the initial model after the loss value converges as the image similarity determination model. The above method determines the positive sample combination and the negative sample combination based on the images generated by the sample image transformation, trains the image similarity determination model through the self-supervised manner, does not need to obtain a large amount of labeled data to train the image similarity determination model, reduces the data acquisition cost, and improves the training efficiency.

[0020] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by means of the structures particularly pointed out in the description and the appended claims.

[0021] In order to make the above-mentioned objects, features and advantages of the present application more apparent, the following preferred embodiments are specifically described with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0023] Figure 1 A flowchart of the training method of the image similarity determination model provided by the embodiment of the present application;

[0024] Figure 2 A flowchart of the image similarity determination method provided by the embodiment of the present application;

[0025] Figure 3 A schematic diagram of sub-image combination provided for an embodiment of the present application;

[0026] Figure 4 A structural schematic diagram of a similarity measurement network provided for an embodiment of the present application;

[0027] Figure 5 A flowchart of another image similarity determination method provided for an embodiment of the present application;

[0028] Figure 6 A structural schematic diagram of a training device of an image similarity determination model provided for an embodiment of the present application;

[0029] Figure 7 A structural schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions and advantages of embodiments of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the drawings. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0031] Comparison between data constitutes a basis in computer science. There are many different comparison methods for different data, for example, Hamming distance is used to compare two binary strings, edit distance is used to compare two texts, or Euclidean distance is used to compare two vectors. Although data comparison is a common data processing method in computer science, similarity measurement closely related to data comparison has certain particularity in the field of images, and is still an open problem that is continuously concerned and discussed. The reason is that, on the one hand, the form of visual image is in a very high-dimensional space and is related to abstract high-level semantics, and on the other hand, the comparison result of image difference or similarity also depends on human subjective perception.

[0032] Therefore, similarity calculation of images needs to solve two problems: (1) reducing the dimension of image space while accurately retaining semantic information. (2) The selected comparison method is as close as possible to human judgment annotation. Selecting a reasonable dimension reduction method can remove irrelevant information while obtaining an objective description, and the reduction of information dimension can greatly improve the efficiency of comparison. The purpose of selecting a suitable comparison method is to ensure the reliability and reasonableness of the result, and to strive to obtain a result consistent with human perception, so as to avoid a result far from the truth.

[0033] In the related art, there are two main categories of image similarity determination methods: one is a traditional scheme, which can directly compare and calculate the original pixel information of the image. Common methods include Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), which are two methods for calculating image degradation. Among them, the peak signal-to-noise ratio is the logarithmic value of the mean square error between the original image and the processed image relative to the maximum pixel value, with a unit of dB, which is a common image quality evaluation index. The structural similarity index includes brightness, contrast and structure, and measures the image similarity by combining the three factors. Structural similarity is more in line with human eye judgment of image quality in image quality measurement.

[0034] The other is to use artificially designed operators to extract features from images and then calculate the distance. Common methods include PHash and AHash. PHash and AHash are both a kind of hash algorithm. Hash algorithm is an algorithm that converts any length of input into a fixed length sequence through hash transformation. The output sequence is the hash value. This conversion is a compression mapping, and the space of the hash value is usually much smaller than the space of the input.

[0035] In the above method, the similarity is calculated directly using the original pixel information, which is relatively convenient, but the noise of the calculated similarity is relatively large due to the low-dimensional characteristics of the pixels. The use of artificial operators to extract features can extract high-dimensional features, but the application scope of the features has great limitations, which affects the robustness of the similarity measurement.

[0036] The other is a deep learning-based scheme. This scheme first pre-trains a convolutional neural network (CNN) model on a large image database with supervised learning, replaces the artificial operator to extract image features, and then calculates the distance between the extracted features to represent the similarity. Convolutional neural network model is a commonly used model in computer vision field, which can efficiently fit the nonlinear relationship between training samples and corresponding labels. This scheme can improve the robustness of the features to a certain extent, but the large-scale supervised data required for pre-training becomes a bottleneck of the scheme. At the same time, it is difficult to measure the similarity by directly using the distance formula, which is close to human perception.

[0037] Therefore, the embodiment of the present application provides an image similarity determination method and a training method and device of a model thereof, which can be applied to various image recognition, image retrieval, image comparison and other scenes.

[0038] Referring to Figure 1 Firstly, a training method of an image similarity determination model provided by an embodiment of the present application is introduced, which comprises the following steps:

[0039] In step S102, training data is generated based on a preset first sample set; the first sample set comprises a plurality of sample images; the training data comprises positive sample combinations and negative sample combinations; each positive sample combination comprises two images, which are both obtained by transformation of a first sample image; each negative sample combination comprises two images, which are respectively obtained by transformation of a first sample image and a second sample image.

[0040] The sample images in the sample set can be a plurality of images in multiple fields, which can be obtained from an image database or an image set collected by a monitoring device or other image collection device. Generally, images with good image quality such as clearness and rich details can be selected as sample images. The differences between different sample images are usually large.

[0041] In order to enable the image similarity determination model to learn the feature differences between two images with low similarity and the feature differences between two images with high similarity, positive sample combinations and negative sample combinations need to be set in the training data. The two images in each positive sample combination have high similarity and can be obtained by transformation of the same sample image; the two images in each negative sample combination have low similarity and are usually obtained by transformation of different sample images.

[0042] The transformation process of the image can include one or more of image processing methods such as rotation, flipping, blurring, noise adding, and twisting. The image processing operations on the two images in the positive sample combination are usually not exactly the same, so that two images that are not exactly the same but have high similarity are obtained. In some special cases, the two images in the positive sample combination can also be the same, in which case the similarity of the two images is the highest. Since the sample images usually have large differences, the two images in the negative sample combination are obtained by transformation of different sample images, so the similarity of the two images in the negative sample combination is low.

[0043] In step S104, the training data is input into an initial model, and the first image features of the images in the positive sample combination and the second image features of the images in the negative sample combination are extracted by the initial model.

[0044] The initial model is usually established based on a neural network or a deep learning model. For example, a convolutional layer in the neural network can extract features of different dimensions in an image by setting parameters such as the size and number of convolution kernels, reduce the dimension of the features extracted by the convolutional layer through a pooling layer, and improve the robustness of the features. By setting the convolution parameters of the convolutional layer and the pooling parameters of the pooling layer in the network structure, and by combining the number, connection order, and the like of the convolutional layer and the pooling layer in various ways, various network structures capable of extracting image features of an image can be obtained. Specifically, an existing neural network structure can be selected to establish the initial model according to requirements, or a network structure applied to the method can be established through experiments to establish the initial model.

[0045] Before the training data is input into the initial model, the images in the training data can be preprocessed, such as cropping the images to a size conforming to the input data of the initial model. The positive sample combination and the negative sample combination can be input into the initial model in turn, and the features of the images in the combination can be extracted by the initial model. The first image features include the features of each image in the positive sample combination, and the second image features include the features of each image in the negative sample combination.

[0046] The initial model is usually used to extract a feature vector of an image, and therefore the first image features and the second image features can be image features represented in the form of a vector. The initial model can also extract a plurality of dimensional feature vectors of an image, and then generate a feature matrix based on the feature vectors, and therefore the first image features and the second image features can also be image features represented in the form of a matrix.

[0047] In step S106, a loss value of the initial model is determined based on the first image features and the second image features.

[0048] Since the similarity between the two images in the positive sample combination is high, the distance between the two features in the first image features output by the initial model should be close. Since the similarity between the two images in the negative sample combination is low, the distance between the two features in the second image features output by the initial model should be far.

[0049] In calculating the loss value, a loss function of a contrast learning algorithm can be considered. When the distance between the two features in the first image features is far and the distance between the two features in the second image features is close, the loss value calculated is large. In specific implementation, other loss functions can also be considered to calculate the loss value, as long as the goal of reducing the feature distance of the images in the positive sample combination and increasing the feature distance of the images in the positive sample combination is achieved.

[0050] Step S108, updating the model parameters of the initial model based on the loss value; continue to perform the step of generating training data based on the preset sample set until the loss value converges, and determine the initial model after the loss value converges as the image similarity determination model.

[0051] After the loss value is calculated, the model parameters of the initial model can be updated based on the loss value. At the same time, the change of the loss value is observed. When the loss value converges, it can be considered that the initial model has been well trained. At this time, the trained initial model can be determined as the image similarity determination model, which can be used in scenarios that require to judge the similarity of two or more images.

[0052] The above-mentioned image similarity determination method generates training data based on a preset first sample set; inputs the training data into an initial model, extracts first image features of images in a positive sample combination and second image features of images in a negative sample combination through the initial model; determines a loss value of the initial model based on the first image features and the second image features; updates the model parameters of the initial model based on the loss value; continues to train the model through the training data until the loss value converges, and determines the initial model after the loss value converges as the image similarity determination model. The above-mentioned method determines the positive sample combination and the negative sample combination based on the images generated by sample image transformation, trains the image similarity determination model through the self-supervised manner, does not need to obtain a large amount of labeled data to train the image similarity determination model, reduces the data acquisition cost, and improves the training efficiency.

[0053] The following embodiments provide an implementation manner of generating training data based on a preset first sample set.

[0054] In the process of generating training data, a first sample image and a second sample image can be first determined from a plurality of sample images; the first image sample is subjected to image processing to obtain a plurality of first transformed images; the second image sample is subjected to image processing to obtain a plurality of second transformed images; the image processing manner can be one or more of a plurality of processing manners such as adding noise, blurring, and changing color parameters, and in specific implementation, different parameters of the processing manner can be set, such as setting different blurring degrees and changing different color parameters, to obtain a plurality of first transformed images and second transformed images.

[0055] After obtaining the first transformed image and the second transformed image, the first image and the second image in the first transformed image can be determined as the positive sample combination, and the third image in the first transformed image and the fourth image in the second transformed image can be determined as the negative sample combination. The third image can be the same as the first image or the second image, or can be another image in the first transformed image.

[0056] The following embodiments provide an implementation of determining a loss value of an initial model based on first image features and second image features.

[0057] The first image features include first features of a first image and second features of a second image in a positive sample combination, and the second image features include third features of a third image and fourth features of a fourth image in a negative sample combination. When calculating the loss value, a loss value function of a contrast learning algorithm can be used to calculate the loss value of the initial model based on the first features, the second features, the third features, and the fourth features.

[0058] When the third image is the same as the first image, a triplet loss function can be used to calculate the loss value, so that the first features of the first image are closer to the second features with higher similarity and farther from the fourth features with lower similarity. A cross-softmax loss function can also be used for calculation. When the loss value is calculated by the loss function, a large number of positive sample combinations and negative sample combinations are usually required, and the loss value is calculated based on the obtained multiple features. A pairwise ranking loss function can also be used to calculate the loss value. The specific setting can be determined according to requirements.

[0059] The embodiments of the present application also provide an image similarity determination method using an image similarity determination model trained by the training method of the image similarity determination model. Figure 2 As shown in the figure, the method comprises the following steps:

[0060] In step S202, a first target image and a second target image are obtained.

[0061] The obtaining process of the first target image and the second target image is related to the application scenario of the method. If the method is applied to an image comparison scenario, the first target image and the second target image are usually images input by a user through a client. If the method is applied to an image retrieval scenario, the first target image can be an image input by a user through a client for retrieval, and the second target image can be an image in a target retrieval image library. If the method is applied to an image recognition scenario, the first target image can be an image input by a user through a client for recognition, and the second target image can be a pre-stored template image.

[0062] Before the first target image and the second target image are determined by the image similarity model, the first target image and the second target image can be pre-processed based on the application scenario. For example, the size of the first target image and the second target image can be adjusted to the same size, and the brightness of the two images can be adjusted to a suitable brightness, etc.

[0063] Step S204, inputting the first target image and the second target image into the pre-trained image similarity determination model, and outputting a first target feature of the first target image and a second target feature of the second target image through the image similarity determination model.

[0064] The image similarity determination model is mainly used to extract image features of images. After the model is trained based on the training method, the image features extracted through the image similarity determination model are more closely related to the images, and the distance between the output image features is smaller when the similarity between the two images is higher.

[0065] Step S206, determining the similarity between the first target image and the second target image based on the first target feature and the second target feature.

[0066] Based on the above discussion, the first target feature and the second target feature can be represented in the form of a vector or a matrix. The distance between the first target feature and the second target feature can be calculated by a pre-set distance calculation formula. The feature distance usually includes Euclidean distance, Chebyshev distance, Minkowski distance, normalized Euclidean distance, Mahalanobis distance, cosine of angle, Hamming distance, etc. Different feature distances may have different aspects of emphasis in the calculation process. One or more feature distance calculation formulas can be selected to calculate the distance between the first target feature and the second target feature, and the distance can be a weighted sum of multiple feature distances.

[0067] The corresponding relationship between the feature distance of the image and the image similarity can be established according to historical experience or experiments, and the similarity between the first target image and the second target image can be finally obtained according to the corresponding relationship and the distance between the first target feature and the second target feature.

[0068] The above method extracts the target features of the first target image and the second target image by training the image similarity determination model in a self-supervised manner, and further determines the similarity between the first target image and the second target image based on the target features. This method does not need to obtain a large amount of labeled data to train the image similarity determination model, reduces the data acquisition cost, and improves the efficiency of determining the image similarity.

[0069] The following embodiments provide an implementation manner for determining the similarity between the first target image and the second target image based on the first target feature and the second target feature.

[0070] In determining the similarity between the first target image and the second target image, a plurality of feature distance formulas can be selected to calculate the distance between the first target feature and the second target feature, for example, the Euclidean distance and the cosine distance between the first target feature and the second target feature. Then, based on the Euclidean distance and the cosine distance between the first target feature and the second target feature, the similarity between the first target image and the second target image can be determined. Specifically, a weight can be set for the two distances to determine the weighted distance between the first target feature and the second target feature, i.e., the weighted sum of the Euclidean distance and the cosine distance, and then according to the corresponding relationship between the feature distance and the image similarity, the similarity between the first target image and the second target image can be calculated according to the weighted distance. The above Euclidean distance and cosine distance can also be replaced by other feature distances, which can be set according to requirements.

[0071] In a specific embodiment, after obtaining the similarity between the first target image and the second target image, the position of the local image with higher similarity in the image also needs to be determined when the similarity is higher.

[0072] In specific implementation, first, it is necessary to determine whether the similarity between the first target image and the second target image is greater than or equal to a target threshold; if so, based on the first target image and the second target image, a plurality of image combinations are generated; the image combination includes a first sub-image of the first target image and a second sub-image corresponding to the first sub-image of the second target image; the position information of the image combination corresponds to the position information of the first sub-image in the first target image, and further, the similarity between each image combination is determined, which can be determined in the following way:

[0073] (1) For each image combination, the first sub-image and the second sub-image are input into a pre-trained feature extraction network, and the multi-dimensional features of the first sub-image and the second sub-image in the image combination are extracted through the feature extraction network.

[0074] Specifically, the first target image can be divided into a plurality of first sub-images, and the second target image can be divided into a plurality of second sub-images corresponding to the first sub-images. When dividing the sub-images, the resolution of the target image can be considered. For example, if the resolution of the target image is 1024*1024, the size of the sub-image is 102*102. A certain ratio is set to form a small window, and then the target image is sampled by sliding the window to form a plurality of sub-images (also referred to as "local region images"). Figure 3 As shown in the first image on the left, four images a, b, c, d correspond to four images A, B, C, D in the second image on the right. The first sub-image and the second sub-image corresponding to the first sub-image are determined as the image combination, and the position information of the first sub-image in the first target image is determined as the position information of the image combination, so Aa, Bb, Cc, Dd are four image combinations.

[0075] The feature extraction network can be an AlexNet network, which can be pre-trained based on an open atlas. The AlexNet network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence. When the multi-dimensional features of the image are needed, the first sub-image and the second sub-image in the image group can be subjected to feature extraction processing through the AlexNet network to obtain a plurality of first sub-features of the first sub-image and a plurality of second sub-features of the second sub-image, respectively. The plurality of first sub-features includes four first sub-features output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively. The plurality of second sub-features includes four second sub-features output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively. Further, the plurality of first sub-features of the first sub-image can be subjected to normalization processing through the second norm to obtain the multi-dimensional features corresponding to the first sub-image, and the plurality of second sub-features of the second sub-image can be subjected to normalization processing through the second norm to obtain the multi-dimensional features corresponding to the second sub-image. The multi-dimensional features can express the characteristic information of the image in different dimensions and are more comprehensive.

[0076] (2) inputting the multi-dimensional features of the first sub-image and the second sub-image into a pre-trained similarity measurement network, processing the multi-dimensional features through the similarity measurement network, and determining the perceptual similarity between the first sub-image and the second sub-image.

[0077] The similarity measurement network can include a plurality of sub-feature networks and an output network. The plurality of sub-feature networks are connected with the output network respectively. The sub-feature network includes a convolutional layer and an average pooling layer. The convolutional layer of each sub-feature network corresponds to a set of convolutional parameters. For example, as shown in FIG. 4, the similarity measurement network includes four sub-feature networks. Figure 4

[0078] When the perceptual similarity between the sub-images is determined through the similarity measurement network, the feature distance between the multi-dimensional features of the first sub-image and the multi-dimensional features of the second sub-image is first calculated. Then, a plurality of distance features of the feature distance are extracted through the plurality of sub-feature networks respectively. The convolutional layer of each sub-feature network of the similarity measurement network corresponds to a set of convolutional parameters, and the dimension of the distance feature corresponds to the convolutional parameters of the sub-feature network extracting the distance feature. Finally, the output network calculates the perceptual similarity between the first sub-image and the second sub-image based on the plurality of distance features.

[0079] The similarity measurement network can be trained in the following manner:

[0080] ​A generate training data based on the preset second sample set; the training data comprises an original sample image, and a first degraded image and a second degraded image corresponding to the original sample image.

[0081] B input the training data into a pre-trained feature extraction network, and extract multi-dimensional features in the original sample image, the first degraded image and the second degraded image through the feature extraction network.

[0082] C input the multi-dimensional features in the original sample image, the first degraded image and the second degraded image into an initial network, and determine a first perceptual similarity between the original sample image and the first degraded image, and a second perceptual similarity between the original sample image and the second degraded image through the initial network.

[0083] D process the first perceptual similarity and the second perceptual similarity through a preset judge, and obtain a perceptual similarity accuracy parameter; the judge is established based on a multi-layer perceptron network.

[0084] E calculate a loss value of a training process based on the first perceptual similarity, the second perceptual similarity and the perceptual similarity accuracy parameter through a cross-entropy loss function.

[0085] F update model parameters of the initial network and the judge based on the loss value; continue to execute the step of generating the training data based on the preset sample set until the loss value converges, and determine the initial network after the loss value converges as a similarity measurement network.

[0086] The judge and the initial network are trained at the same time, so the model parameters of the initial network and the judge need to be updated. When the loss value converges, it is considered that the initial network has been trained well, and the trained initial network can be determined as the similarity measurement network.

[0087] (3) determine a similarity parameter of the first target image and the second target image based on the position information of the image combination and the perceptual similarity corresponding to the image combination; the similarity parameter is used to represent the local similarity of the first target image and the second target image.

[0088] The embodiment of the application also provides another image similarity determination method, which is implemented based on the method shown in the embodiment. Figure 1 The method explores how to accurately represent image information and how to reliably calculate the similarity between images.

[0089] The flow of the method is specifically as follows: first, two images that need to be subjected to similarity detection are selected, and after the legality of the images is detected, such as whether the format of the images is a preset format, if the images are legal, the two images are taken as a first target image and a second target image. Then, feature extraction and preliminary calculation of similarity are performed based on the whole image, in the case of meeting a certain similarity threshold, the similarity is more accurately calculated for local features using perceptual similarity, and finally the calculated similarity and the similar region range are returned.

[0090] In the above method, the image similarity determination model is used for feature extraction of the whole image. Since the contrast learning method is used in the process of training the image similarity determination model, the training process is free from the bottleneck of supervised data, so that the model can directly learn how to describe image information from a large number of unlabeled images. Therefore, the model has strong information extraction ability, and the extracted image features are more accurate.

[0091] In the above method, the similarity of the local image is calculated, so that the comparison process is more reliable. The neural network is used to perform similarity fitting learning on a small amount of labeled data set, instead of simply relying on distance formula for calculation, so that the result of similarity calculation is more consistent with human cognition.

[0092] The above method uses a multi-stage image similarity calculation method to automatically detect in the whole and local two dimensions without any manual operation, so that the comparison method is more simple and flexible. And it has strong expandability. The method can be used immediately, and can be quickly applied to any image comparison task, such as image similarity scoring, image search, and copy detection.

[0093] The flowchart of the above method is shown in Figure 5 The main steps include image global and local feature extraction, global and local similarity calculation, etc. The feature extraction model is derived from the contrast learning training stage, and the local similarity calculation method is derived from the perceptual similarity learning training stage. The key technologies involved in the method include:

[0094] (1) Image global feature extraction module: This module is responsible for extracting features in the whole image. The features output by this module will be the main information for calculating similarity and input into the global similarity calculation module. This module is the core module of the whole system, and the features represented by it need to meet:

[0095] a. Accuracy: can accurately depict the main information of the image, focusing on the grasp of key feature points of the main body of the image. These feature points are on the one hand a description of the main body of the image itself, and on the other hand can be clearly distinguished from the main body of other images.

[0096] b Briefly, the high-dimensional feature space is converted to a low-dimensional feature space to achieve dimensionality reduction of features. Through feature dimensionality reduction, on the one hand, the efficiency of computer operation can be ensured, and on the other hand, effective information can be maximized to weaken or eliminate irrelevant information.

[0097] (2) Global similarity calculation module: This module receives the features of two images as input information, and obtains the distance as the representation of similarity after corresponding calculation according to the selected distance formula, so as to preliminarily measure the similarity of the two images.

[0098] (3) Image local feature extraction module: This module is responsible for extracting the features of the local region of the image. The features output by this module will be the main information for calculating the local similarity and will be input into the local similarity calculation module.

[0099] (4) Local similarity calculation module: This module receives the features of the local subgraph of the image as input information, and no longer relies on the distance formula but uses the method of perceptual similarity to accurately measure the similarity between the subgraphs. The similarity calculation module needs to meet:

[0100] a Rationality: The similarity measurement result obtained by calculation needs to be consistent with human cognition itself, and the result of similar images should have certain factual basis so as to obtain reasonable explanation;

[0101] b distinguishability, different subjective similarity degrees should be objectively reflected in the numerical value of similarity measurement, and more similar images should have higher numerical value of similarity, while less similar images should have lower numerical value of similarity;

[0102] In combination Figure 5 , the technical details and main implementation modes involved in the four important modules in the method are introduced as follows.

[0103] The image global feature extraction module is used to extract the features of the whole image as the input of the global similarity calculation. In the training process, the contrast learning method is used to pre-train the convolutional neural network using a large amount of unsupervised data, and then the pre-trained model is used to complete the feature extraction of the whole image, as follows:

[0104] Training stage: The EfficientNet network structure is used as the feature extractor in the model, and the self-supervised training is performed on the public DISC21 dataset based on the contrast learning method. The images in the dataset are transformed to obtain different transformed image versions, which are used as training samples. The transformations include various deformation and degradation methods such as rotation, flipping, blurring, noise addition, etc. Random combinations of these deformations and degradations are applied to the images. The corresponding formula is as follows:

[0105]

[0106]

[0107] where Aug(;θ m ) represents the transformation combination function under the parameter θ m , I + and I - represent two independent images respectively, and represent the new images generated by the combination of the images I + under different combination transformations, where and are positive sample pairs, and and are negative sample pairs. After obtaining the positive and negative sample data, the model is trained using the contrastive loss, and the loss function is as follows:

[0108]

[0109]

[0110] where z i , z k are image features generated by the EfficientNet network structure and the Phead network used for training. zi is the image feature of the image of the positive sample, and zk is the image feature of the image of the non-sample. The Phead network can generate image features that make the EfficientNet network structure converge faster during training based on the image features extracted by the EfficientNet network structure, to improve the training speed.

[0111] The EfficientNet network pre-trained using contrastive learning performs end-to-end feature extraction on the images, and the corresponding formula is represented as:

[0112] f e =M efficient (I,θ e )

[0113] where M efficent (,θ e ) represents the feature extraction network under the parameter θ e , where represents an input image with a size of N x M, represents a feature vector with a length of d after network extraction. In this method, we set N and M to 224, and d to 256.

[0114] The role of the global similarity calculation module is mainly to use the features obtained by the image global feature extraction module to calculate the distance between the features using the distance formula to represent the distance between the whole images, thereby obtaining a preliminary similarity measurement to provide a basis for subsequent fine measurement. Specifically, Euclidean distance and cosine distance can be used to measure feature distance. The formula is:

[0115]

[0116]

[0117] The image local feature extraction module is used to calculate the local features of the image. First, the image can be sliced, and the main structure of the AlexNet network is used as the feature extractor of the sliced subgraph. This extractor has fast operation speed and high accuracy. We directly use the pre-trained network weight on the ImageNet database for initialization, extract the output of the last four layers of the network, and add l2 norm for normalization processing as the description feature of the image. The formula is:

[0118] f a =Norm(M Ale x(I;θ a ))

[0119] On the basis of global similarity calculation, the local feature is used to calculate the similarity, which can more accurately find the similar range and area between images and images, and provide more accurate comparison results. For the local similarity calculation module, instead of using the formula to calculate the feature distance, a multi-layer convolution and pooling CNN model is used to let the network model automatically learn the similarity judgment standard, thereby obtaining a perceptual similarity comparable to human cognitive standard. The details of this method are as follows:

[0120] Training phase: based on the public database 2AFC, we use the designed similarity measurement network to calculate the distance between two images with different degradation degrees and the original image. At the same time, we also design a judge M g to judge whether the calculated distance is accurate. The judge and the similarity measurement network are trained at the same time, and the cross-entropy loss is used to constrain the training process. The formula is:

[0121] d1=M sim (k1,k;θ s );d2=M sim (k2,k;θ s )

[0122] label=M g (d1,d2;θ g ),

[0123] wherein k is a feature of the original image, k1 is a feature of an image generated based on degradation of the original image, and k2 is a feature of another image generated based on degradation of the original image; d1 is a similarity between the original image and the image corresponding to k1 output by the similarity measurement network; and d2 is a similarity between the original image and the image corresponding to k2 output by the similarity measurement network.

[0124] A test phase: using the trained similarity measurement network M sim (;θ s ), inputting the features of the two subgraphs, and automatically calculating the corresponding similarity d e (0, 1), wherein the closer the similarity result is to 1, the more similar the two graphs are, and the closer the result is to 0, the greater the difference is.

[0125] The above method has wide application scenarios, mainly including three aspects: a. image similarity calculation alone; b. search based on image form; and c. copy detection based on image content. In terms of the method itself, the similarity detection function provided by the method is extremely close to human perception, and can be directly used in comparison of two images. Meanwhile, the method has high flexibility and expansibility, and can be directly extended from comparison of a single image to comparison of multiple images or even a database. For example, the method can be introduced in search, thereby realizing the function of "searching by image" based on image search. The method can also be used in image copy detection, and by detecting the similarity between a user-uploaded image and an existing library image, whether the user-uploaded image has copy, tampering or other infringement problems can be detected.

[0126] For the above method embodiment, the training device of the image similarity determination model provided by the embodiment of the present application shown in Figure 6 includes:

[0127] The training data generation module 602 is configured to generate training data based on a preset first sample set. The first sample set includes a plurality of sample images. The training data includes positive sample combinations and negative sample combinations. The positive sample combinations include two images, which are both obtained by transformation of a first sample image. The negative sample combinations include two images, which are respectively obtained by transformation of the first sample image and a second sample image.

[0128] The feature extraction module 604 is configured to input the training data into an initial model, and extract first image features of the images in the positive sample combinations and second image features of the images in the negative sample combinations by the initial model.

[0129] The first loss value determination module 606 is configured to determine a loss value of the initial model based on the first image features and the second image features.

[0130] The parameter updating module 608 is configured to update the model parameters of the initial model based on the loss value; continue to perform the step of generating the training data based on the preset sample set until the loss value converges, and determine the initial model after the loss value converges as the image similarity determination model.

[0131] The image similarity determination apparatus includes the training data generation module, the initial model, the loss value determination module, and the parameter updating module.

[0132] The training data generation module is further configured to determine the first sample image and the second sample image from the plurality of sample images; perform image processing on the first sample image to obtain a plurality of first transformed images; perform image processing on the second sample image to obtain a plurality of second transformed images; determine the first image and the second image in the first transformed images as the positive sample combination; and determine a third image in the first transformed images and a fourth image in the second transformed images as the negative sample combination.

[0133] The first image features include first features of the first image and second features of the second image in the positive sample combination; the second image features include third features of the third image and fourth features of the fourth image in the negative sample combination; and the first loss value determination module is further configured to calculate the loss value of the initial model based on the first features of the first image and the second features of the second image in the positive sample combination, and the third features of the third image and the fourth features of the fourth image in the negative sample combination, through a loss function of a contrastive learning algorithm.

[0134] For the method embodiment, the present embodiment provides an image similarity determination apparatus, which includes:

[0135] The target image acquisition module 702 is configured to acquire the first target image and the second target image.

[0136] The feature output module 704 is configured to input the first target image and the second target image into a pre-trained image similarity determination model, output first target features of the first target image and second target features of the second target image through the image similarity determination model, and train the image similarity model through the training method of the image similarity determination model.

[0137] The similarity determination module 706 is configured to determine the similarity of the first target image and the second target image based on the first target features and the second target features.

[0138] The device extracts target features of the first target image and the second target image through training of the image similarity determination model in a self-supervised manner, and further determines the similarity of the first target image and the second target image based on the target features. This method does not need to obtain a large amount of labeled data to train the image similarity determination model, reduces the data acquisition cost, and improves the efficiency of determining the image similarity.

[0139] The similarity determination module is configured to: calculate the Euclidean distance and the cosine distance between the first target features and the second target features; and determine the similarity of the first target image and the second target image based on the Euclidean distance and the cosine distance between the first target features and the second target features.

[0140] The device further includes: a threshold determination module configured to determine whether the similarity of the first target image and the second target image is greater than or equal to a target threshold; an image combination generation module configured to, if yes, generate a plurality of image combinations based on the first target image and the second target image; the image combination includes a first sub-image of the first target image and a second sub-image of the second target image corresponding to the first sub-image; the position information of the image combination corresponds to the position information of the first sub-image in the first target image; a multi-dimensional feature extraction module configured to, for each image combination, input the first sub-image and the second sub-image into a pre-trained feature extraction network, and extract multi-dimensional features of the first sub-image and the second sub-image in the image combination through the feature extraction network; a perception similarity determination module configured to input the multi-dimensional features of the first sub-image and the second sub-image into a pre-trained similarity measurement network, process the multi-dimensional features through the similarity measurement network, and determine a perception similarity of the first sub-image and the second sub-image; a similarity parameter determination module configured to determine a similarity parameter of the first target image and the second target image based on the position information of the image combination and the perception similarity corresponding to the image combination; and the similarity parameter is used to represent a local similarity of the first target image and the second target image.

[0141] The image combination generation module is further configured to divide the first target image into a plurality of first sub-images, divide the second target image into a plurality of second sub-images corresponding to the first sub-images, and determine the first sub-image and the second sub-image corresponding to the first sub-image as the image combination, and determine position information of the first sub-image in the first target image as position information of the image combination.

[0142] The feature extraction network comprises an AlexNet network; the AlexNet network comprises a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first full connection layer, a second full connection layer and a third full connection layer connected in sequence; the multi-dimensional feature extraction module is further configured to perform feature extraction processing on the first sub-image and the second sub-image in the image combination by the AlexNet network, and obtain a plurality of first sub-features of the first sub-image and a plurality of second sub-features of the second sub-image respectively; the plurality of first sub-features are output by the fifth convolutional layer, the first full connection layer, the second full connection layer and the third full connection layer respectively; the second sub-features are output by the fifth convolutional layer, the first full connection layer, the second full connection layer and the third full connection layer respectively; the plurality of first sub-features of the first sub-image are normalized by the second norm to obtain the multi-dimensional feature corresponding to the first sub-image; and the plurality of second sub-features of the second sub-image are normalized by the second norm to obtain the multi-dimensional feature corresponding to the second sub-image.

[0143] The device further comprises a similarity measurement network training module configured to generate training data based on the preset second sample set; the training data comprises an original sample image, and a first degraded image and a second degraded image corresponding to the original sample image; input the training data into the pre-trained feature extraction network, extract the multi-dimensional features in the original sample image, the first degraded image and the second degraded image by the feature extraction network; input the multi-dimensional features in the original sample image, the first degraded image and the second degraded image into the initial network, determine the first perceptual similarity between the original sample image and the first degraded image, and the second perceptual similarity between the original sample image and the second degraded image by the initial network; process the first perceptual similarity and the second perceptual similarity by the preset judge to obtain a perceptual similarity accuracy parameter; the judge is established based on a multi-layer perceptron network; calculate the loss value of the training process based on the first perceptual similarity, the second perceptual similarity and the perceptual similarity accuracy parameter by the cross-entropy loss function; update the model parameters of the initial network and the judge based on the loss value; continue to execute the step of generating training data based on the preset sample set until the loss value converges, and determine the initial network after the loss value converges as the similarity measurement network.

[0144] The similarity measurement network includes a plurality of sub-feature networks and an output network; the plurality of sub-feature networks are connected with the output network respectively; the sub-feature network includes a convolution layer and an average pooling layer; the convolution layer of each sub-feature network corresponds to a set of convolution parameters; the similarity measurement network training module is further configured to: calculate a feature distance between the multi-dimensional features of the first sub-image and the multi-dimensional features of the second sub-image; extract a plurality of distance features of the feature distance through the plurality of sub-feature networks respectively; the dimensions of the distance features correspond to the convolution parameters of the sub-feature network that extracts the distance features; and calculate the perceptual similarity of the first sub-image and the second sub-image based on the plurality of distance features through the output network.

[0145] The embodiment also provides an electronic device including a processor and a memory. The memory stores machine executable instructions capable of being executed by the processor. The processor executes the machine executable instructions to implement the training method of the image similarity determination model or the image similarity determination method.

[0146] The training method of the image similarity determination model is as follows: generating training data based on a preset first sample set; the first sample set includes a plurality of sample images; the training data includes positive sample combinations and negative sample combinations; each positive sample combination includes two images, which are both transformed from a first sample image; each negative sample combination includes two images, which are respectively transformed from a first sample image and a second sample image; inputting the training data into an initial model, extracting first image features of the images in the positive sample combinations and second image features of the images in the negative sample combinations through the initial model; determining a loss value of the initial model based on the first image features and the second image features; updating model parameters of the initial model based on the loss value; continuing to execute the step of generating training data based on the preset sample set until the loss value converges, and determining the initial model after the loss value converges as the image similarity determination model.

[0147] The method determines positive sample combinations and negative sample combinations based on images transformed from sample images, and trains the image similarity determination model in a self-supervised manner, without the need to obtain a large amount of labeled data to train the image similarity determination model, thereby reducing the data acquisition cost and improving the training efficiency.

[0148] Optionally, the step of generating training data based on the preset first sample set includes: determining a first sample image and a second sample image from the plurality of sample images; performing image processing on the first image sample to obtain a plurality of first transformed images; performing image processing on the second image sample to obtain a plurality of second transformed images; determining a first image and a second image in the first transformed images as a positive sample combination; and determining a third image in the first transformed images and a fourth image in the second transformed images as a negative sample combination.

[0149] Optionally, the first image feature includes a first feature of a first image and a second feature of a second image in the positive sample combination, and the second image feature includes a third feature of a third image and a fourth feature of a fourth image in the negative sample combination; the step of determining the loss value of the initial model based on the first image feature and the second image feature includes: calculating the loss value of the initial model by a loss function of a contrastive learning algorithm based on the first feature of the first image and the second feature of the second image in the positive sample combination, and the third feature of the third image and the fourth feature of the fourth image in the negative sample combination.

[0150] The image similarity determination method is specifically as follows: obtaining a first target image and a second target image; inputting the first target image and the second target image into a pre-trained image similarity determination model, outputting a first target feature of the first target image and a second target feature of the second target image by the image similarity determination model; the image similarity model is trained by the training method of the image similarity determination model; and determining the similarity between the first target image and the second target image based on the first target feature and the second target feature.

[0151] The method extracts the target features of the first target image and the second target image by training the image similarity determination model in a self-supervised manner, and further determines the similarity between the first target image and the second target image based on the target features. This method does not need to obtain a large amount of labeled data to train the image similarity determination model, thereby reducing the data acquisition cost and improving the efficiency of determining the image similarity.

[0152] Optionally, the step of determining the similarity between the first target image and the second target image based on the first target feature and the second target feature includes: calculating the Euclidean distance and the cosine distance between the first target feature and the second target feature; and determining the similarity between the first target image and the second target image based on the Euclidean distance and the cosine distance between the first target feature and the second target feature.

[0153] Optionally, the method further comprises: determining whether the similarity between the first target image and the second target image is greater than or equal to a target threshold; if yes, generating a plurality of image combinations based on the first target image and the second target image; the image combination comprises a first sub-image of the first target image and a second sub-image of the second target image corresponding to the first sub-image; the position information of the image combination corresponds to the position information of the first sub-image in the first target image; for each image combination, inputting the first sub-image and the second sub-image into a pre-trained feature extraction network, extracting multi-dimensional features of the first sub-image and the second sub-image in the image combination through the feature extraction network; inputting the multi-dimensional features of the first sub-image and the second sub-image into a pre-trained similarity measurement network, processing the multi-dimensional features through the similarity measurement network to determine the perceptual similarity between the first sub-image and the second sub-image; determining a similarity parameter between the first target image and the second target image based on the position information of the image combination and the perceptual similarity corresponding to the image combination; the similarity parameter is used to represent the local similarity between the first target image and the second target image.

[0154] Optionally, the step of generating a plurality of image combinations based on the first target image and the second target image comprises: dividing the first target image into a plurality of first sub-images, and dividing the second target image into a plurality of second sub-images corresponding to the first sub-images; determining the first sub-image and the second sub-image corresponding to the first sub-image as an image combination, and determining the position information of the first sub-image in the first target image as the position information of the image combination.

[0155] Optionally, the feature extraction network comprises an AlexNet network; the AlexNet network comprises a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence; the step of extracting multi-dimensional features of the first sub-image and the second sub-image in the image combination through the feature extraction network comprises: performing feature extraction processing on the first sub-image and the second sub-image in the image combination through the AlexNet network to obtain a plurality of first sub-features of the first sub-image and a plurality of second sub-features of the second sub-image, respectively; the plurality of first sub-features are output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively; the second sub-features are output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively; performing normalization processing on the plurality of first sub-features of the first sub-image through the second norm to obtain the multi-dimensional features corresponding to the first sub-image; performing normalization processing on the plurality of second sub-features of the second sub-image through the second norm to obtain the multi-dimensional features corresponding to the second sub-image.

[0156] Optionally, the similarity measurement network is trained in the following manner: based on a preset second sample set, training data is generated; the training data includes an original sample image, and a first degraded image and a second degraded image corresponding to the original sample image; the training data is input into the pre-trained feature extraction network, and multi-dimensional features in the original sample image, the first degraded image and the second degraded image are extracted by the feature extraction network; the multi-dimensional features in the original sample image, the first degraded image and the second degraded image are input into the initial network, and a first perceptual similarity between the original sample image and the first degraded image and a second perceptual similarity between the original sample image and the second degraded image are determined by the initial network; the first perceptual similarity and the second perceptual similarity are processed by a preset judge to obtain a perceptual similarity accuracy parameter; the judge is established based on a multi-layer perceptron network; a loss value of the training process is calculated based on the first perceptual similarity, the second perceptual similarity and the perceptual similarity accuracy parameter by a cross-entropy loss function; model parameters of the initial network and the judge are updated based on the loss value; the step of generating the training data based on the preset sample set is continuously performed until the loss value converges, and the initial network after the loss value converges is determined as the similarity measurement network.

[0157] Optionally, the similarity measurement network includes a plurality of sub-feature networks and an output network; the plurality of sub-feature networks are connected with the output network respectively; the sub-feature network includes a convolution layer and an average pooling layer; the convolution layer of each sub-feature network corresponds to a set convolution parameter; the step of determining the perceptual similarity of the first sub-image and the second sub-image by processing the multi-dimensional features by the similarity measurement network includes: calculating a feature distance between the multi-dimensional features of the first sub-image and the multi-dimensional features of the second sub-image; extracting a plurality of distance features of the feature distance by the plurality of sub-feature networks respectively; the dimension of the distance feature corresponds to the convolution parameter of the sub-feature network extracting the distance feature; the perceptual similarity of the first sub-image and the second sub-image is calculated based on the plurality of distance features by the output network.

[0158] Referring to Figure 7 The electronic device shown in the figure includes a processor 100 and a memory 101 storing machine executable instructions executable by the processor 100, and the processor 100 executes the machine executable instructions to implement the training method of the image similarity determination model or the image similarity determination method.

[0159] Further, Figure 7 The electronic device shown in the figure further includes a bus 102 and a communication interface 103, and the processor 100, the communication interface 103 and the memory 101 are connected through the bus 102.

[0160] The memory 101 can include a random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 102 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the figure to represent the system network element and at least one other network element, but it does not mean that there is only one bus or one type of bus.

[0161] The processor 100 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 100 or the instructions in the form of software. The processor 100 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiment of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101, and combines the hardware to complete the steps of the method of the above embodiment.

[0162] The embodiment also provides a machine readable storage medium, the machine readable storage medium stores machine executable instructions, and the machine executable instructions cause the processor to implement the training method of the image similarity determination model when the processor calls and executes the machine executable instructions.

[0163] The embodiment of the present application provides a kind of image similarity determination model training method, device and electronic equipment, including the computer readable storage medium of program code storage, the instruction included in the program code can be used to execute the above-mentioned image similarity determination model training method or image similarity determination method described in the above method embodiment.

[0164] The above-mentioned image similarity determination model training method is as follows: based on the first sample set preset, generate training data;The first sample set includes multiple sample images;Training data includes positive sample combination and negative sample combination;Positive sample combination includes two images, which are obtained by transforming the first sample image;Negative sample combination includes two images, which are obtained by transforming the first sample image and the second sample image respectively;The first image feature of the image in the positive sample combination and the second image feature of the image in the negative sample combination are extracted by inputting the training data into the initial model through the initial model;Determine the loss value of the initial model based on the first image feature and the second image feature;Update the model parameters of the initial model based on the loss value;Continue to execute the step of generating training data based on the preset sample set until the loss value converges, and determine the initial model after the loss value converges as the image similarity determination model.

[0165] The above-mentioned method determines the positive sample combination and the negative sample combination based on the images generated by transforming the sample images, and trains the image similarity determination model by the self-supervised mode, without the need to obtain a large amount of labeled data to train the image similarity determination model, thereby reducing the data acquisition cost and improving the training efficiency.

[0166] Optionally, the step of generating training data based on the first sample set preset includes: determining the first sample image and the second sample image from the multiple sample images;Image processing is carried out on the first image sample to obtain multiple first transformed images;Image processing is carried out on the second image sample to obtain multiple second transformed images;The first image and the second image in the first transformed image are determined as the positive sample combination;The third image in the first transformed image and the fourth image in the second transformed image are determined as the negative sample combination.

[0167] Optionally, the first image feature includes the first feature of the first image and the second feature of the second image in the positive sample combination;The second image feature includes the third feature of the third image and the fourth feature of the fourth image in the negative sample combination;The step of determining the loss value of the initial model based on the first image feature and the second image feature includes: based on the first feature of the first image and the second feature of the second image in the positive sample combination, and the third feature of the third image and the fourth feature of the fourth image in the negative sample combination, the loss value of the initial model is calculated by the loss function of the contrast learning algorithm.

[0168] The image similarity determination method is specifically as follows: a first target image and a second target image are obtained; the first target image and the second target image are input into a pre-trained image similarity determination model, and a first target feature of the first target image and a second target feature of the second target image are output by the image similarity determination model; the image similarity model is trained by the training method of the image similarity determination model; and the similarity of the first target image and the second target image is determined based on the first target feature and the second target feature.

[0169] The method extracts target features of the first target image and the second target image by training the image similarity determination model in a self-supervised manner, and further determines the similarity of the first target image and the second target image based on the target features. This method does not need to obtain a large amount of labeled data to train the image similarity determination model, thereby reducing the data acquisition cost and improving the efficiency of determining the image similarity.

[0170] Optionally, the step of determining the similarity of the first target image and the second target image based on the first target feature and the second target feature includes: calculating the Euclidean distance and the cosine distance between the first target feature and the second target feature; and determining the similarity of the first target image and the second target image based on the Euclidean distance and the cosine distance between the first target feature and the second target feature.

[0171] Optionally, the method further includes: determining whether the similarity of the first target image and the second target image is greater than or equal to a target threshold; if yes, generating a plurality of image combinations based on the first target image and the second target image; the image combination includes a first sub-image of the first target image and a second sub-image of the second target image corresponding to the first sub-image; the position information of the image combination corresponds to the position information of the first sub-image in the first target image; for each image combination, the first sub-image and the second sub-image are input into a pre-trained feature extraction network, and multi-dimensional features of the first sub-image and the second sub-image in the image combination are extracted by the feature extraction network; the multi-dimensional features of the first sub-image and the second sub-image are input into a pre-trained similarity measurement network, and the multi-dimensional features are processed by the similarity measurement network to determine the perceptual similarity of the first sub-image and the second sub-image; the similarity parameter of the first target image and the second target image is determined based on the position information of the image combination and the perceptual similarity corresponding to the image combination; and the similarity parameter is used to represent the local similarity of the first target image and the second target image.

[0172] Optionally, the step of generating the plurality of image combinations based on the first target image and the second target image comprises: dividing the first target image into a plurality of first sub-images, and dividing the second target image into a plurality of second sub-images corresponding to the first sub-images; determining the first sub-image and the second sub-image corresponding to the first sub-image as the image combination, and determining position information of the first sub-image in the first target image as position information of the image combination.

[0173] Optionally, the feature extraction network comprises an AlexNet network; the AlexNet network comprises a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence; the step of extracting the multi-dimensional features of the first sub-image and the second sub-image in the image combination by the feature extraction network comprises: performing feature extraction processing on the first sub-image and the second sub-image in the image combination by the AlexNet network to obtain a plurality of first sub-features of the first sub-image and a plurality of second sub-features of the second sub-image, respectively; the plurality of first sub-features are output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively; the second sub-features are output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively; the plurality of first sub-features of the first sub-image are normalized by the second norm to obtain the multi-dimensional features corresponding to the first sub-image; and the plurality of second sub-features of the second sub-image are normalized by the second norm to obtain the multi-dimensional features corresponding to the second sub-image.

[0174] Optionally, the similarity measurement network is trained in the following manner: based on the preset second sample set, training data is generated; the training data comprises an original sample image, and a first degraded image and a second degraded image corresponding to the original sample image; the training data is input into the pre-trained feature extraction network, and multi-dimensional features in the original sample image, the first degraded image, and the second degraded image are extracted by the feature extraction network; the multi-dimensional features in the original sample image, the first degraded image, and the second degraded image are input into the initial network, and a first perceptual similarity between the original sample image and the first degraded image and a second perceptual similarity between the original sample image and the second degraded image are determined by the initial network; the first perceptual similarity and the second perceptual similarity are processed by a preset judge to obtain a perceptual similarity accuracy parameter; the judge is established based on a multi-layer perceptron network; a loss value of the training process is calculated based on the first perceptual similarity, the second perceptual similarity, and the perceptual similarity accuracy parameter by a cross-entropy loss function; model parameters of the initial network and the judge are updated based on the loss value; the step of generating the training data based on the preset sample set is continuously performed until the loss value converges, and the initial network after the loss value converges is determined as the similarity measurement network.

[0175] Optionally, the similarity measurement network comprises a plurality of sub-feature networks and an output network; the plurality of sub-feature networks are connected with the output network respectively; the sub-feature network comprises a convolution layer and an average pooling layer; the convolution layer of each sub-feature network corresponds to a set of convolution parameters; the step of determining the perceptual similarity between the first sub-image and the second sub-image by processing the multi-dimensional features through the similarity measurement network comprises: calculating the feature distance between the multi-dimensional features of the first sub-image and the multi-dimensional features of the second sub-image; extracting a plurality of distance features of the feature distance through the plurality of sub-feature networks respectively; the dimension of the distance feature corresponds to the convolution parameters of the sub-feature network extracting the distance feature; and calculating the perceptual similarity between the first sub-image and the second sub-image based on the plurality of distance features through the output network.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0177] In addition, in the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0178] The functions described above, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the present application which essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of software products, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage program codes.

[0179] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0180] Finally, it should be noted that the above embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, and are not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent substitutions for some technical features; and these modifications, changes or substitutions do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image similarity determination method characterized by, The method comprises the following steps: obtaining a first target image and a second target image; inputting the first target image and the second target image into a pre-trained image similarity determination model, outputting a first target feature of the first target image and a second target feature of the second target image through the image similarity determination model; determining the similarity of the first target image and the second target image based on the first target feature and the second target feature; judging whether the similarity of the first target image and the second target image is greater than or equal to a target threshold value; if yes, generating a plurality of image combinations based on the first target image and the second target image; the image combination comprises a first sub-image of the first target image and a second sub-image of the second target image corresponding to the first sub-image; the position information of the image combination corresponds to the position information of the first sub-image in the first target image; the first sub-image is obtained by sliding window sampling on the first target image; the second sub-image is obtained by sliding window sampling on the second target image; for each image combination, inputting the first sub-image and the second sub-image into a pre-trained feature extraction network, and extracting multi-dimensional features of the first sub-image and the second sub-image in the image combination through the feature extraction network; inputting the multi-dimensional features of the first sub-image and the second sub-image into a pre-trained similarity measurement network, processing the multi-dimensional features through the similarity measurement network, and determining the perceptual similarity of the first sub-image and the second sub-image; based on the position information of the image combination and the perceptual similarity corresponding to the image combination, determining the similarity parameter of the first target image and the second target image; the similarity parameter is used to represent the local similarity of the first target image and the second target image.

2. The method of claim 1, wherein, The image similarity determination model is trained in the following way: based on a pre-set first sample set, generate training data; the first sample set comprises a plurality of sample images; the training data comprises positive sample combinations and negative sample combinations; the positive sample combination comprises two images, which are obtained by transforming a first sample image; the negative sample combination comprises two images, which are obtained by transforming a first sample image and a second sample image, respectively; input the training data into an initial model, extract the first image feature of the image in the positive sample combination and the second image feature of the image in the negative sample combination through the initial model; based on the first image feature and the second image feature, determine the loss value of the initial model; update the model parameters of the initial model based on the loss value; continue to execute the step of generating training data based on the pre-set sample set until the loss value converges, and determine the initial model after the loss value converges as the image similarity determination model.

3. The method of claim 2, wherein, The step of generating training data based on the pre-set first sample set comprises: determining a first sample image and a second sample image from the plurality of sample images; performing image processing on the first image sample to obtain a plurality of first transformed images; performing image processing on the second image sample to obtain a plurality of second transformed images; determining a first image and a second image in the first transformed images as a positive sample combination; determining a third image in the first transformed images and a fourth image in the second transformed images as a negative sample combination.

4. The method of claim 2, wherein, The first image features include first features of the first image and second features of the second image in the positive sample combination; the second image features include third features of the third image and fourth features of the fourth image in the negative sample combination; Based on the first image features and the second image features, the step of determining the loss value of the initial model includes: Based on the first features of the first image and the second features of the second image in the positive sample combination, and the third features of the third image and the fourth features of the fourth image in the negative sample combination, the loss value of the initial model is calculated through the loss function of the contrastive learning algorithm.

5. The method of claim 1, wherein, Based on the first target features and the second target features, the step of determining the similarity of the first target image and the second target image includes: Calculate the Euclidean distance and cosine distance between the first target features and the second target features; Based on the Euclidean distance and cosine distance between the first target features and the second target features, determine the similarity of the first target image and the second target image.

6. The method of claim 1, wherein, Based on the first target image and the second target image, the step of generating a plurality of image combinations includes: Divide the first target image into a plurality of first sub-images, and divide the second target image into a plurality of second sub-images corresponding to the first sub-images; Determine the first sub-image and the second sub-image corresponding to the first sub-image in the image combination as the image combination, and determine the position information of the first sub-image in the first target image as the position information of the image combination.

7. The method of claim 1, wherein, The feature extraction network includes an AlexNet network; the AlexNet network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a first fully connected layer, a second fully connected layer and a third fully connected layer connected in turn; The step of extracting the multi-dimensional features of the first sub-image and the second sub-image in the image combination through the feature extraction network includes: The first sub-image and the second sub-image in the image combination are processed by the AlexNet network for feature extraction, and a plurality of first sub-features of the first sub-image and a plurality of second sub-features of the second sub-image are obtained respectively; a plurality of first sub-features are output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer and the third fully connected layer respectively; the second sub-features are output by the fifth convolutional layer, the first fully connected layer, the second fully connected layer and the third fully connected layer respectively; The plurality of first sub-features of the first sub-image are normalized by the second norm to obtain the multi-dimensional features corresponding to the first sub-image; The second sub-features of the second sub-image are normalized by a second norm to obtain multi-dimensional features corresponding to the second sub-image.

8. The method of claim 1, wherein, The similarity measurement network is trained in the following manner: Based on a preset second sample set, training data is generated; the training data includes an original sample image, and a first degraded image and a second degraded image corresponding to the original sample image; The multi-dimensional features in the original sample image, the first degraded image and the second degraded image are input into an initial network, and the first perception similarity between the original sample image and the first degraded image and the second perception similarity between the original sample image and the second degraded image are determined by the initial network; The first perception similarity and the second perception similarity are processed by a preset judge to obtain a perception similarity accuracy parameter; the judge is established based on a multi-layer perceptron network; Based on the first perception similarity, the second perception similarity and the perception similarity accuracy parameter, a loss value of the training process is calculated by a cross-entropy loss function; Based on the loss value, the model parameters of the initial network and the judge are updated; the step of generating training data based on a preset sample set is continuously executed until the loss value converges, and the initial network after the loss value converges is determined as the similarity measurement network. The similarity measurement network includes a plurality of sub-feature networks and an output network; a plurality of the sub-feature networks are connected with the output network respectively; the sub-feature network includes a convolution layer and an average pooling layer; the convolution layer of each sub-feature network corresponds to a set convolution parameter; 9. The method of claim 1, wherein, The step of determining the perception similarity of the first sub-image and the second sub-image by processing the multi-dimensional features by the similarity measurement network includes: The feature distance between the multi-dimensional features of the first sub-image and the multi-dimensional features of the second sub-image is calculated; A plurality of distance features of the feature distance are extracted by a plurality of the sub-feature networks respectively; the dimension of the distance feature corresponds to the convolution parameter of the sub-feature network extracting the distance feature; The perception similarity of the first sub-image and the second sub-image is calculated based on a plurality of the distance features by the output network. It includes:

10. An image similarity determination apparatus characterized by comprising: The target image acquisition module is used for acquiring a first target image and a second target image; The feature output module is used for inputting the first target image and the second target image into a pre-trained image similarity determination model, outputting a first target feature of the first target image and a second target feature of the second target image by the image similarity determination model; the image similarity model is trained by the training method of the image similarity determination model in any one of claims 1-3. ​ The similarity determination module is configured to determine a similarity between the first target image and the second target image based on the first target feature and the second target feature. The device further comprises: The threshold determination module is configured to determine whether the similarity between the first target image and the second target image is greater than or equal to a target threshold. The image combination generation module is configured to generate a plurality of image combinations based on the first target image and the second target image if the similarity between the first target image and the second target image is greater than or equal to the target threshold; each of the image combinations comprises a first sub-image of the first target image and a second sub-image of the second target image corresponding to the first sub-image; the position information of each of the image combinations corresponds to the position information of the first sub-image in the first target image; the first sub-image is obtained by performing sliding window sampling on the first target image; and the second sub-image is obtained by performing sliding window sampling on the second target image. The multi-dimensional feature extraction module is configured to input the first sub-image and the second sub-image of each of the image combinations into a pre-trained feature extraction network, and extract multi-dimensional features of the first sub-image and the second sub-image in each of the image combinations through the feature extraction network. The perceptual similarity determination module is configured to input the multi-dimensional features of the first sub-image and the second sub-image into a pre-trained similarity measurement network, process the multi-dimensional features through the similarity measurement network, and determine a perceptual similarity between the first sub-image and the second sub-image. The similarity parameter determination module is configured to determine a similarity parameter between the first target image and the second target image based on the position information of each of the image combinations and the perceptual similarity corresponding to each of the image combinations; the similarity parameter is used to represent a local similarity between the first target image and the second target image.

11. An electronic device, comprising: The processor and the memory are included, and the memory stores machine executable instructions capable of being executed by the processor; the processor executes the machine executable instructions to implement the image similarity determination method in any one of claims 1-9.

12. A machine-readable storage medium, characterized in that, The machine readable storage medium stores machine executable instructions; when the machine executable instructions are called and executed by the processor, the machine executable instructions cause the processor to implement the image similarity determination method in any one of claims 1-9.

Images