Image similarity measurement method, system and equipment robust to geometric transformation

Abstract

The invention discloses an image similarity measurement method, system and device robust to geometric transformation. According to the invention, a Hartley pooling-based graph classification neural network framework with robustness to geometric transformation is designed, the step length of a convolution layer with the step length of 2 in ResNet is changed into 1, and a Hartley pooling layer with the same dimension reduction effect is connected behind each group of convolution layers. According to the method, a new designed framework is used for training an image classification model, output of a feature layer of the trained image classification model is used as input features during calculation of image similarity measurement, and a shallow image similarity calculation neural network is trained. The image similarity measurement scheme provided by the invention is closer to human visual judgment, and the robustness of geometric transformation is obviously superior to the current leading technical level.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to an image similarity measurement method, system and device that are robust to geometric transformation. Background Art

[0002] Image similarity measurement is the basis for the implementation of many network security and privacy services. A typical application scenario is the Client-side Scanning (CSS) used by Apple and Facebook to monitor the spread of Child Sexual Abuse Material (CSAM) in chat software. The specific operating mechanism of CSS is: the operator establishes a database of illegal images; every time a user sends an image in the chat software, the system uses this image as input and performs a perceptual hash-based search on the database; if this process finds a match with a similarity above the threshold, it will trigger further review by the operator. The "perceptual hash proximity" here is a commonly used image similarity measurement.

[0003] The performance of mechanical traditional image similarity metrics, such as PNSR, SSIM, etc., is not enough to support such applications - what we desire is to approach the "perceptual distance" of human vision for image similarity judgment. The most widely recognized image similarity metric that meets this requirement is Learned Perceptual Image Patch Similarity (LPIPS). As a newer image similarity metric based on neural networks, its approximation of human visual perception is significantly better than traditional similarity metrics. Its value is relatively robust under pixel-by-pixel editing such as color adjustment and blurring that do not change the pixel position. However, it has obvious fragility to geometric transformations such as rotation, flipping, and translation. The existence of this shortcoming is not ideal for many of its potential application scenarios, because humans can easily obtain the information contained in the original image from the image that has been geometrically transformed. Summary of the invention

[0004] The purpose of the present invention is to provide an image similarity measurement method, system and device that are robust to geometric transformations in view of the deficiencies of the prior art.

[0005] According to a first aspect of the present specification, a method for measuring image similarity that is robust to geometric transformation is provided, comprising:

[0006] Build a graph classification neural network framework that is robust to geometric transformations, change the stride of the convolutional layer in ResNet from 2 to 1, and add a Hartley pooling layer after each set of convolutional layers to achieve the same dimensionality reduction effect;

[0007] Based on the image classification dataset, the graph classification model is trained using the constructed graph classification neural network framework;

[0008] The training results of the feature layer of the graph classification model are transferred to the image similarity measurement calculation task.

[0009] Furthermore, the calculation process of the Hartley pooling layer includes: performing a discrete Hartley transform on each channel of the input value, centering the low-frequency data through a translation operation, cutting out the high-frequency data according to the dimensionality reduction requirements, and then performing an inverse translation to release the centering, and performing an inverse discrete Hartley transform to complete the dimensionality reduction.

[0010] Furthermore, the training results of the feature layer of the graph classification model are transferred to the image similarity measurement calculation task, specifically:

[0011] Extract several feature layers from the trained image classification model, and use the two images whose similarity is to be calculated as input. For the output value of each layer, first normalize it along the number of channels, then calculate the distance between the features of the two inputs at each pixel, use the Hartley pooling layer to unify the height and width of all features, and then splice them along the dimension of the number of channels;

[0012] A computer vision convolutional neural network is constructed which takes the shape of the spliced ​​image as input as the image similarity calculation neural network.

[0013] Furthermore, the first linear rectification function layer, layer1, layer2, layer3, and layer4, a total of five feature layers, are extracted from the trained graph classification model and merged as the initial layer.

[0014] Furthermore, the image similarity calculation neural network includes a convolutional layer, a Hartley pooling layer and a fully connected layer connected in sequence.

[0015] Furthermore, the image similarity calculation neural network is recorded as ,in is the parameter to be trained, are the parameters obtained through training; for the two images whose similarity is to be calculated , define the intermediate function and similarity measure function ,in Based on The resulting stitched image;

[0016] Prepare the form training samples, where is the reference image, for and Between The relative similarity weight of , optimize the following binary cross entropy loss function: .

[0017] According to a second aspect of the present specification, a system for measuring image similarity that is robust to geometric transformation is provided. The system is used to implement the image similarity measurement method according to the first aspect. The system comprises:

[0018] The graph classification neural network framework construction module is used to build a graph classification neural network framework that is robust to geometric transformations. The stride of the convolutional layer in ResNet is changed from 2 to 1, and a Hartley pooling layer with the same dimensionality reduction effect is connected after each group of convolutional layers.

[0019] The graph classification model training module is used to train the graph classification model based on the image classification dataset using the constructed graph classification neural network framework that is robust to geometric transformations;

[0020] The image similarity measurement module is used to migrate the training results of the feature layer of the graph classification model to the image similarity measurement calculation task.

[0021] According to a third aspect of the present specification, an electronic device is provided, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the image similarity measurement method as described in the first aspect.

[0022] According to a fourth aspect of the present specification, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the image similarity measurement method as described in the first aspect is implemented.

[0023] According to a fifth aspect of the present specification, a computer program product is provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the image similarity measurement method as described in the first aspect is implemented.

[0024] The beneficial effects of the present invention are as follows: the present invention designs a graph classification neural network framework based on Hartley pooling that is robust to geometric transformations, changes the step size of the convolution layer with a step size of 2 in ResNet to 1, and connects a Hartley pooling layer with the same dimensionality reduction effect after each group of convolution layers. The present invention uses the designed new framework to train a graph classification model, uses the output of the feature layer of the trained graph classification model as the input feature when calculating the image similarity measure, and trains a shallow image similarity calculation neural network. The image similarity measurement scheme provided by the present invention is closer to human visual judgment, and its robustness to geometric transformations is significantly better than the current leading technology level. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 The overall flow chart of an image similarity measurement method robust to geometric transformation is shown as an exemplary embodiment;

[0027] Figure 2 The figure is a schematic diagram showing an implementation principle of an image similarity measurement method that is robust to geometric transformations according to an exemplary embodiment;

[0028] Figure 3 A diagram showing the computational principle of the Hartley pooling layer in a graph classification neural network framework as an exemplary embodiment;

[0029] Figure 4 The structure diagram of an image similarity measurement system that is robust to geometric transformation is shown as an exemplary embodiment;

[0030] Figure 5 The figure is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0032] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0033] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0034] The present invention proposes a method, system and device for measuring image similarity that is robust to geometric transformation. Figure 1 As shown in the figure, it can be summarized as follows: design a graph classification neural network framework based on Hartley pooling that is robust to geometric transformations; based on the image classification dataset, use the designed new framework to train the graph classification model; and transfer the training results of the feature layer of the graph classification model to the image similarity measurement calculation task.

[0035] The principle behind the scheme of the present invention is a phenomenon widely discovered by scholars in the field: the feature layer of a neural network model trained for a specific computer vision task often obtains feature quantities that can be widely transferred to other computer vision tasks. However, in order to achieve the purpose of the present invention being robust to geometric transformations, a framework that is sufficiently robust to geometric transformations needs to be used when training graph classification models. To this end, the present invention introduces Hartley pooling based on the Discrete Hartley Transform into the neural network framework to enhance the robustness of the neural network framework to geometric transformations. On the one hand, the dimensionality reduction mechanism for the input is an essential part of the neural network framework for computer vision applications, but it is also a step that causes the loss of geometric information of the input. On the other hand, the Hartley transform is a frequency domain transform, and for naturally generated images, the low-frequency part usually contains most of the meaningful information while the high-frequency part contains mostly noise. Therefore, the present invention focuses on this law to design a dimensionality reduction mechanism that is sufficiently robust to geometric transformations. For example Figure 2 As shown, the specific implementation process of the image similarity measurement method that is robust to geometric transformation proposed by the present invention is described in detail below.

[0036] 101. Build a graph classification neural network framework that is robust to geometric transformations.

[0037] According to the task characteristics of the present invention, the most commonly used neural network framework ResNet in the field of computer vision is optimized to obtain a new framework with better robustness to geometric perturbations of input values. ResNet originally achieved gradual dimensionality reduction of input values ​​through a series of convolutional layers with a stride of 2. The present invention changes the stride of the convolutional layer with an original stride of 2 to 1, and connects a Hartley pooling layer with the same dimensionality reduction effect after each group of convolutional layers. The intuitive principle of Hartley pooling is as follows Figure 3 shown.

[0038] Specifically, the Hartley pooling layer performs the following calculations. is the tensor representation of the input value of the Hartley pooling layer, where are the number of channels, height, and width of the image, respectively. Perform a discrete Hartley transform on each channel l:

[0039]

[0040]

[0041] Secondly, according to the usual frequency domain analysis, the low frequency is in the center and the high frequency is at the edge through translation:

[0042]

[0043] in It means round down, and mod means remainder.

[0044] Then according to the dimensionality reduction requirements (here we assume that the dimension to be reduced to ) to trim off the data corresponding to high frequencies:

[0045]

[0046] Then perform an inverse translation to remove the centering expression:

[0047]

[0048] Finally, the inverse discrete Hartley transform is performed (here the inverse transform corresponds to the previous Hartley transform, so the denominator in the trigonometric function is Rather than ):

[0049]

[0050]

[0051]

[0052] 102. Use the new framework constructed in step 101 to train a graph classification model.

[0053] The new framework constructed in step 101 is recorded as ResNetR2GT (ResNet Robust to Geometric Transformation). In this step, based on the image classification data set, the new framework is used to train the graph classification model and determine its parameter values. Specifically, the optimizer used in this embodiment is the Stochastic Gradient Descent algorithm, and the hyperparameters are the learning rate that decreases by 1 / 10 every 30 iterations (epochs) from the initial value of 0.1 and the momentum of 0.9. The loss function uses the cross entropy loss function.

[0054] 103. Extract the feature layer from the graph classification model trained in step 102 and merge them as the initial layer, and build a shallow image similarity calculation neural network on top of it.

[0055] After obtaining the specific parameter values ​​of ResNetR2GT in step 102, the outputs of several feature layers are extracted as input features for calculating the image similarity metric. Specifically, this embodiment extracts the outputs of the first linear rectification function (ReLU) layer of ResNetR2GT and the subsequent five layers, namely layer1, layer2, layer3, and layer4 (the names of the layers here follow the conventional naming method of the ResNet framework). Represents the output values ​​of these five layers when the input is x. The output value is a 3-dimensional tensor (ie, number of channels * height * width). The number of channels, height, and width of the output are , , .

[0056] set up are the two input images whose similarity is to be calculated. For each layer, normalization is first performed along the number of channels:

[0057]

[0058] Then the distance between the two input features is calculated at each pixel:

[0059]

[0060] Note here The output is still the number of channels, height, and width. , , 3D tensor of . Hartley pooling is used to pool all features The height and width of are unified for subsequent splicing operations. The features after size unification are recorded as Specifically, the height and width of all features are unified to a size that does not exceed the minimum height and minimum width of all features. In this embodiment, Hartley pooling is used to unify the (height, width) of the features into :

[0061]

[0062] in Refers to the output (height, width) as The Hartley pooling layer. Therefore The output is the number of channels, height, and width. , , 3D tensor of . Since Hartley pooling can better preserve the geometric information of the data, the robustness of this step to the geometric transformation of the input is maximized. These layers are concatenated along the dimension of the number of channels to form a 3D tensor with the number of channels, height, and width. , , A 3D tensor .

[0063] because The present invention constructs a computer vision convolutional neural network with a conventional image data shape (i.e., number of channels * height * width) as an image similarity calculation neural network. Specifically, the following image similarity calculation neural network is constructed: ,in Are the parameters that need to be trained:

[0064] The first layer: convolutional layer. This embodiment uses The convolution layer has a step size of 1;

[0065] The second layer: Hartley pooling layer. This embodiment uses an output size of The Hartley pooling layer;

[0066] The third layer: a fully connected layer with an output dimension of 1.

[0067] set up are the parameters obtained through training. Since the similarity measurement requires two images As an input function, the present invention defines the following intermediate function And the final similarity measurement function :

[0068]

[0069]

[0070] The reason for defining an intermediate function in addition to the similarity measurement function is that the present invention utilizes the intermediate function To train the parameters .

[0071] 104. Use the image similarity judgment dataset to train the image similarity calculation neural network.

[0072] The entire calculation formula for the image similarity measure of the present invention has been described in step 103. The only remaining technical details are the parameters training method.

[0073] First, the data format of the image similarity judgment dataset used for training is explained. A single training sample must have of the form, where are three images, r is the reference image, for and The relative similarity weight between them for r: Close to 0 means Compare More like r; Close to 1 indicates Compare More like r; If it is around 0.5, it means and When collecting data, the present invention adopts the same group Let multiple human subjects judge and Which one is more like r, and then take the relative frequency of different judgments as value.

[0074] During the training process, a logistic function layer is superimposed on the image similarity calculation neural network given in step 103 to obtain a value range of Specifically, let is a training sample, As defined in step 103. Let the prediction function The form is as follows:

[0075]

[0076] Then optimize the following binary cross entropy loss function:

[0077]

[0078] The optimizer used for training is ADAM (Adaptive Moment Estimation), and the hyperparameters are the learning rate with an initial value of 0.0001 and the exponential decay rate of the first-order moment estimate of 0.5. and an exponential decay rate of the second moment estimate of 0.999 .

[0079] On the other hand, an embodiment of the present invention provides a geometric transformation robust image similarity measurement system 2000, which is used to implement the above-mentioned geometric transformation robust image similarity measurement method, such as Figure 4 As shown, the system includes:

[0080] Graph classification neural network framework construction module 2001 is used to build a graph classification neural network framework that is robust to geometric transformations. The stride of the convolution layer in ResNet is changed from 2 to 1, and a Hartley pooling layer with the same dimensionality reduction effect is connected after each group of convolution layers.

[0081] The graph classification model training module 2002 is used to train a graph classification model based on an image classification data set using a graph classification neural network framework that is robust to geometric transformations and is constructed by the graph classification neural network framework construction module 2001;

[0082] The image similarity measurement module 2003 is used to transfer the training results of the feature layer of the graph classification model trained by the graph classification model training module 2002 to the image similarity measurement calculation task.

[0083] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0084] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without creative work.

[0085] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned image similarity measurement method that is robust to geometric transformation. Figure 5 As shown, a hardware structure diagram of any device with data processing capability for the method for measuring image similarity robust to geometric transformation provided by an embodiment of the present invention is shown, except Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0086] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the image similarity measurement method that is robust to geometric transformations as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0087] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0088] It will be appreciated that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

[0089] The above is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above disclosed methods and technical contents without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A method for measuring image similarity that is robust to geometric transformations, characterized in that: The method comprises: Build a graph classification neural network framework that is robust to geometric transformations, change the stride of the convolutional layer in ResNet from 2 to 1, and add a Hartley pooling layer after each set of convolutional layers to achieve the same dimensionality reduction effect; Based on the image classification dataset, the graph classification model is trained using the constructed graph classification neural network framework; The training results of the feature layer of the graph classification model are transferred to the image similarity measurement calculation task.

2. The method according to claim 1, characterized in that The calculation process of the Hartley pooling layer includes: performing a discrete Hartley transform on each channel of the input value, centering the low-frequency data through a translation operation, cutting off the high-frequency data according to the dimensionality reduction requirements, and then performing an inverse translation to release the centering, and performing an inverse discrete Hartley transform to complete the dimensionality reduction.

3. The method according to claim 1, characterized in that The training results of the feature layer of the graph classification model are transferred to the image similarity measurement calculation task, specifically: Extract several feature layers from the trained image classification model, and use the two images whose similarity is to be calculated as input. For the output value of each layer, first normalize it along the number of channels, then calculate the distance between the features of the two inputs at each pixel, use the Hartley pooling layer to unify the height and width of all features, and then splice them along the dimension of the number of channels; A computer vision convolutional neural network is constructed which takes the shape of the spliced ​​image as input as the image similarity calculation neural network.

4. The method according to claim 3, characterized in that The first linear rectification function layer, layer1, layer2, layer3, and layer4, a total of five feature layers, are extracted from the trained graph classification model and merged as the initial layer.

5. The method according to claim 3, characterized in that: The image similarity calculation neural network includes a convolutional layer, a Hartley pooling layer and a fully connected layer which are connected in sequence.

6. The method according to claim 3, characterized in that The image similarity calculation neural network is denoted as ,in is the parameter to be trained, are the parameters obtained through training; for the two images whose similarity is to be calculated , define the intermediate function and similarity measure function ,in Based on The resulting stitched image; Prepare the form training samples, where is the reference image, for and Between The relative similarity weight of , optimize the following binary cross entropy loss function: .

7. An image similarity measurement system that is robust to geometric transformations, characterized in that: Used to implement the image similarity measurement method according to any one of claims 1 to 6, the system comprises: The graph classification neural network framework construction module is used to build a graph classification neural network framework that is robust to geometric transformations. The stride of the convolutional layer in ResNet is changed from 2 to 1, and a Hartley pooling layer with the same dimensionality reduction effect is connected after each group of convolutional layers. The graph classification model training module is used to train the graph classification model based on the image classification dataset using the constructed graph classification neural network framework that is robust to geometric transformations; The image similarity measurement module is used to migrate the training results of the feature layer of the graph classification model to the image similarity measurement calculation task.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the image similarity measurement method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image similarity measurement method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the image similarity measurement method according to any one of claims 1 to 6 is implemented.

Images