Image protection method and related device

Abstract

The application discloses an image protection method, comprising: obtaining a source image and a random image, the source image comprising a target protection object, and the random image not comprising the target protection object; initializing an initial protection image according to the source image; calculating a first feature distance between the initial protection image and the random image, a second feature distance between the initial protection image and the source image, and an apparent distance; calculating a loss function according to the first feature distance, the second feature distance, and the apparent distance; and iteratively updating the initial protection image based on the loss function by using a back propagation algorithm to obtain a protection image of the target protection object. The technical solution provided by the application can protect the target object in the image, avoid the image being counterfeited or stolen for illegal attack, and ensure information security. The application also discloses an image protection device, equipment and a computer readable storage medium, which have the above beneficial effects.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image protection method and related equipment. Background Technology

[0002] With the development of deep learning technology, artificial intelligence applications based on deep learning (such as facial recognition, speech recognition, natural language processing, and autonomous driving) have permeated all aspects of social life, profoundly changing human production and lifestyles. At the same time, artificial intelligence technology also carries risks and challenges, threatening human life and property. For example, autonomous vehicles may fail to detect pedestrians or other vehicles in time, leading to traffic accidents; malicious attackers may use forged / stolen photos or videos to deceive facial recognition systems; and social groups may use forged videos for false propaganda. Preventing the various risks arising from artificial intelligence applications has become a major concern for all sectors of society.

[0003] Among the many risks, the unauthorized acquisition and use of facial information seriously threatens the personal and property safety of individuals. Therefore, owners of facial images (usually themselves) wish to protect their images and prevent unauthorized use. For example, owners do not want their images to be acquired by commercial companies for advertising or training facial recognition models; even more so, they do not want their images to be stolen and used to attack their personal bank accounts.

[0004] Therefore, how to protect the target objects in an image, prevent the image from being forged or stolen for illegal attack purposes, and ensure information security is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide an image protection method that can protect the target object in the image, prevent the image from being forged or stolen for illegal attack purposes, and ensure information security. Another purpose of this application is to provide an image protection device, equipment, and computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0006] In a first aspect, this application provides an image protection method, the method comprising:

[0007] Acquire a source image and a random image, wherein the source image includes the target protected object and the random image does not include the target protected object;

[0008] An initial protection image is generated based on the source image;

[0009] Calculate the first feature distance between the initial protected image and the random image, the second feature distance between the initial protected image and the source image, and the apparent distance;

[0010] The loss function is calculated based on the first feature distance, the second feature distance, and the apparent distance.

[0011] Based on the loss function, the initial protected image is iteratively updated using the backpropagation algorithm to obtain a protected image of the target protected object.

[0012] Optionally, calculating the first feature distance between the initial protected image and the random image includes:

[0013] The initial protected image and the random image are processed using a face recognition model to obtain a first feature vector and a second feature vector.

[0014] Calculate the cosine distance between the first feature vector and the second feature vector to obtain the first feature distance.

[0015] Optionally, calculating the second feature distance between the initial protected image and the source image includes:

[0016] The source image is processed using the face recognition model to obtain a third feature vector;

[0017] Calculate the cosine distance between the first feature vector and the third feature vector to obtain the second feature distance.

[0018] Optionally, the face recognition model is a neural network model based on ResNet50.

[0019] Optionally, calculating the apparent distance between the initial protected image and the source image includes:

[0020] The apparent distance is obtained by calculating the F-norm of the initial protected image and the source image.

[0021] Optionally, calculating the apparent distance between the initial protected image and the source image includes:

[0022] The initial protected image and the source image are input into an image classification network for processing to obtain the apparent distance.

[0023] Optionally, the image classification network is a VGG-16-based neural network model.

[0024] Optionally, before generating the initial protected image based on the source image, the method further includes:

[0025] Based on the input specifications of the face recognition model, the source image and the random image are standardized respectively to obtain standardized source image and standardized random image;

[0026] The standardized source image and the standardized random image are in the image format specified by the face recognition model.

[0027] Optionally, the source image is normalized to obtain the normalized source image, including:

[0028] Perform type recognition on the source image to determine the image type;

[0029] The source image is standardized by using the processing strategy corresponding to the image type to obtain the standardized source image.

[0030] Optionally, the source image is normalized to obtain the normalized source image, including:

[0031] Construct an affine transformation matrix using the source image and image samples;

[0032] The source image is converted into the normalized source image using the affine transformation matrix.

[0033] Optionally, constructing the affine transformation matrix using the source image and image samples includes:

[0034] Obtain the first coordinate information of the preset feature points of the target protected object in the source image;

[0035] Obtain the second coordinate information of preset feature points of sample objects in each of the image samples;

[0036] The affine transformation matrix is ​​calculated using the first coordinate information and each of the second coordinate information.

[0037] Optionally, calculating the affine transformation matrix using the first coordinate information and each of the second coordinate information includes:

[0038] The affine transformation matrix is ​​obtained by calculating the first coordinate information and each of the second coordinate information using the least squares estimation algorithm.

[0039] Secondly, this application also discloses an image protection device, the device comprising:

[0040] The acquisition module is used to acquire a source image and a random image, wherein the source image includes the target protected object and the random image does not include the target protected object;

[0041] The processing module is used to initialize and generate an initial protection image based on the source image;

[0042] The first calculation module is used to calculate the first feature distance between the initial protected image and the random image, and the second feature distance and apparent distance between the initial protected image and the source image;

[0043] The second calculation module is used to calculate a loss function based on the first feature distance, the second feature distance, and the apparent distance.

[0044] The update module is used to iteratively update the initial protected image based on the loss function using the backpropagation algorithm to obtain a protected image of the target protected object.

[0045] Thirdly, this application also discloses an image protection device, comprising:

[0046] Memory, used to store computer programs;

[0047] A processor for executing the computer program to implement any of the image protection methods described above.

[0048] Fourthly, this application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the image protection methods described above.

[0049] The image protection method provided in this application includes: acquiring a source image and a random image, wherein the source image includes a target object to be protected, and the random image does not include the target object to be protected; generating an initial protection image based on the source image; calculating a first feature distance between the initial protection image and the random image, a second feature distance and an apparent distance between the initial protection image and the source image; calculating a loss function based on the first feature distance, the second feature distance and the apparent distance; and iteratively updating the initial protection image using a backpropagation algorithm based on the loss function to obtain a protection image for the target object to be protected.

[0050] By applying the technical solution provided in this application, the feature distance between the protected image and the source image and the feature distance between the protected image and the random image are considered simultaneously. Based on the face recognition model and the backpropagation algorithm, the feature distance between the protected image and the random image is compressed, while the feature distance between the protected image and the source image is expanded. This results in the final protected image being far away from the source image and close to the random image. In other words, the final protected image containing the target object is closer to the random image that does not contain the target object, and far away from the original image containing the target object. At this time, the target object recognition system will have difficulty identifying the target object in the protected image, thereby achieving privacy protection of the target object. This can effectively prevent the image from being forged or stolen for illegal attack purposes and ensure information security.

[0051] The image protection device, equipment, and computer-readable storage medium provided in this application also have the above-mentioned technical effects, and will not be described in detail here. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the prior art and the embodiments of this application, the accompanying drawings used in the description of the prior art and the embodiments of this application will be briefly introduced below. Of course, the accompanying drawings described below with respect to the embodiments of this application are only a part of the embodiments in this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort, and such other drawings also fall within the protection scope of this application.

[0053] Figure 1 A flowchart illustrating an image protection method provided in this application;

[0054] Figure 2 A schematic flowchart of an image protection device provided in this application;

[0055] Figure 3 This is a schematic diagram of the structure of an image protection device provided in this application. Detailed Implementation

[0056] The core of this application is to provide an image protection method that can securely protect target objects in an image, prevent the image from being forged or stolen for use in illegal attacks, and ensure information security. Another core aspect of this application is to provide an image protection device, equipment, and computer-readable storage medium, all of which have the aforementioned beneficial effects.

[0057] To provide a clearer and more complete description of the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0058] This application provides an image protection method.

[0059] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an image protection method provided in this application, which may include the following steps S101 to S105.

[0060] S101: Obtain a source image and a random image. The source image includes the target protected object, while the random image does not include the target protected object.

[0061] This step aims to acquire source images and random images. Source images are images containing the target object, while random images are images that do not contain the target object. The target object is the object in the image that needs information protection, which is generally facial information.

[0062] In the implementation process, when it is necessary to protect the image of a certain target object, a source image containing the target object and a random image not containing the target object can be obtained first. The source of these two images does not affect the implementation of this technical solution. They can be image information acquired by an image acquisition device, image information directly input by the user at the device front end, or image information retrieved from an image database. This application does not limit this. In addition, the image types of the source image and the random image are not unique. For example, they can be three-channel color images or single-channel grayscale images. This application also does not limit this.

[0063] Among them, a random image refers to an image that does not contain the target object. Therefore, it can be an image that contains any target object other than the current target object (such as other face images), an image that does not contain any target object (such as an image that does not contain any face information), or an automatically generated random image.

[0064] S102: Generate an initial protected image based on the source image;

[0065] This step aims to generate an initial protected image based on the source image. This initial protected image is the first image obtained containing the target object and protection information about the target object. By updating this initial protected image, the final protected image (described in S105) can be obtained, which is the image that achieves protection of the target object in the image. From a visual perspective, the protected image generated by the image protection method provided in this application is similar to the source image. Therefore, using the source image to initialize the initial protected image can initialize the initial protected image near the target value and accelerate the convergence of the algorithm. Using other initialization methods (such as random initialization) to generate the initial protected image may cause the algorithm to diverge.

[0066] S103: Calculate the first feature distance between the initial protected image and the random image, the second feature distance between the initial protected image and the source image, and the apparent distance;

[0067] This step aims to calculate the feature distance and apparent distance. The feature distance refers to the feature distance between the initial protected image and the random image, as well as the feature distance between the initial protected image and the source image. The apparent distance refers to the apparent distance between the initial protected image and the source image. After obtaining the source image, the random image, and the initial protected image, the first feature distance, the second feature distance, and the apparent distance can be calculated based on the feature information in each image.

[0068] S104: Calculate the loss function based on the first feature distance, the second feature distance, and the apparent distance;

[0069] This step aims to calculate the loss function. During the process, after obtaining the first feature distance between the initial protected image and the random image, the second feature distance between the initial protected image and the source image, and the apparent distance, these three values ​​can be substituted into a pre-defined loss function calculation formula to obtain the overall loss function.

[0070] S105: Based on the loss function, the backpropagation algorithm is used to iteratively update the initial protected image to obtain a protected image of the target protected object.

[0071] This step aims to update the initial protected image to obtain the final protected image of the target object. In this process, after calculating the overall loss function, the initial protected image is iteratively updated based on this loss function and the backpropagation algorithm to obtain the protected image. This protected image is image data containing the target object and strong protective information about it.

[0072] As can be seen, the image protection method provided in this application takes into account both the feature distance between the protected image and the source image and the feature distance between the protected image and the random image. Based on the face recognition model and the backpropagation algorithm, it compresses the feature distance between the protected image and the random image and expands the feature distance between the protected image and the source image. This makes the final generated protected image closer to the random image and further away from the source image. In other words, the final generated protected image containing the target object is closer to the random image that does not contain the target object and further away from the original image containing the target object. At this time, the target object recognition system will have difficulty identifying the target object in the protected image, thereby achieving privacy protection of the target object. This can effectively prevent the image from being forged or stolen for illegal attack purposes and ensure information security.

[0073] In one embodiment of this application, calculating the first feature distance between the initial protected image and the random image may include the following steps:

[0074] The initial protected image and the random image are processed using a face recognition model to obtain the first feature vector and the second feature vector.

[0075] Calculate the cosine distance between the first eigenvector and the second eigenvector to obtain the first eigendistance.

[0076] This application provides a method for calculating a first feature distance between an initial protected image and a random image, wherein the cosine distance between the initial protected image and the random image is used as the first feature distance. In the implementation process, the initial protected image and the random image can be input into a face recognition model for processing. Through forward propagation of the neural network, feature vectors of the two images are obtained, namely the aforementioned first feature vector and second feature vector. Then, the cosine distance between the two feature vectors is calculated to obtain the first feature distance.

[0077] The face recognition model is a pre-created neural network learning model, pre-stored in the corresponding storage space, and can be directly called when needed. It should be noted that in the feature space of the face recognition model, the distance between the features of the initial protected image and the features of the random image is specified to be less than the distance between the features of the initial protected image and the features of the source image. Of course, the specific network type used in the face recognition model does not affect the implementation of this technical solution; it can be selected by technicians according to actual needs. This application does not limit this. In one possible implementation, the face recognition model is a ResNet50-based neural network model, that is, using ResNet50 as the backbone network.

[0078] In one embodiment of this application, calculating the second feature distance between the initial protected image and the source image may include the following steps:

[0079] The source image is processed using a face recognition model to obtain a third feature vector;

[0080] Calculate the cosine distance between the first and third eigenvectors to obtain the second eigendistance.

[0081] This application provides a method for calculating the second feature distance between an initial protected image and a source image, wherein the cosine distance between the initial protected image and the source image is used as the second feature distance. The specific implementation process can be referred to the previous embodiment, and will not be repeated here.

[0082] In one embodiment of this application, calculating the apparent distance between the initial protected image and the source image may include the following steps:

[0083] The apparent distance is obtained by calculating the F-norm of the initial protected image and the source image.

[0084] This application provides a method for calculating the apparent distance between an initial protected image and a source image, namely, calculating the apparent distance between the initial protected image and the source image based on the F-norm calculation formula.

[0085] In one embodiment of this application, calculating the apparent distance between the initial protected image and the source image may include the following steps:

[0086] The initial protected image and the source image are input into an image classification network for processing to obtain the apparent distance.

[0087] This application provides another method for calculating the apparent distance between the initial protected image and the source image. This method can be implemented based on a pre-created image classification network. After obtaining the initial protected image and the source image, they can be input into the image classification network for processing. The distance between the outputs of the image classification network is the apparent distance between the two images.

[0088] Among them, the image classification network is a pre-created neural network, which is pre-stored in the corresponding storage space and can be directly called when used. Similar to the image recognition model mentioned above, its network type is not unique and can be set by technicians according to actual needs. In one possible implementation, the image classification network can be a neural network model based on the VGG-16 network (an image classification network).

[0089] In one embodiment of this application, before generating the initial protected image based on the source image, the following steps may be included:

[0090] Based on the input specifications of the face recognition model, the source image and the random image are standardized respectively to obtain standardized source image and standardized random image;

[0091] Among them, the image formats of the standardized source image and the standardized random image are the image formats specified by the face recognition model.

[0092] The image protection method provided in this application can standardize both the source image and the random image before inputting them into the face recognition model. This standardization process aims to convert the image formats of both images into the image format specified by the face recognition model, so as to facilitate subsequent model-based image processing and ensure the accuracy of the image processing results. Therefore, the standardization process can be performed with reference to the input specifications of the face recognition model, which indicate the format requirements of the face recognition model for the input image.

[0093] The specific implementation of the formatting process is not unique and can be set by technical personnel according to actual needs. For example, it can be an image transformation method such as image rotation, image scaling, or affine transformation. This application does not limit this.

[0094] In one embodiment of this application, standardizing the source image to obtain a standardized source image may include the following steps:

[0095] Perform type recognition on the source image to determine the image type;

[0096] The source image is standardized by using the processing strategy corresponding to the image type to obtain a standardized source image.

[0097] As mentioned above, the image types of source images and random images are not unique; for example, they can be three-channel color images or single-channel grayscale images. Based on this, different normalization processing methods can be adopted for different types of source images to adapt to them. Therefore, preprocessing strategies for different image types can be created in advance. After obtaining the source image, its image type can be identified first to determine its type, and then the source image can be normalized using the processing strategy corresponding to that image type to obtain a normalized source image.

[0098] Of course, the standardization process for random images can also be implemented in the above manner to obtain standardized random images, which will not be elaborated here.

[0099] In one embodiment of this application, standardizing the source image to obtain a standardized source image may include the following steps:

[0100] Construct an affine transformation matrix using the source image and image samples;

[0101] The source image is converted into a normalized source image using an affine transformation matrix.

[0102] This application provides an implementation method for standardization processing, namely image standardization processing based on an affine transformation matrix. In the implementation process, an affine transformation matrix is ​​first constructed using the source image and image samples. This affine transformation matrix is ​​specific to the source image; different images correspond to different affine transformation matrices. Furthermore, the source image is directly converted into a standardized source image using this affine transformation matrix.

[0103] Image samples refer to sample data containing other types of target objects (different from the target objects in the source image). Taking face images as an example, the source image contains the target face information that needs to be protected, while the sample image also contains face information, but it is completely different from the face information in the source image (not belonging to the same face). In one possible implementation, the face information in the sample image can be average face information generated based on a neural network or downloaded from the network. It is understandable that the number of image samples is not unique; the more image samples there are, the higher the accuracy of the constructed affine transformation matrix.

[0104] Of course, the standardization process for random images can also be implemented in the above manner to obtain standardized random images, which will not be elaborated here.

[0105] In one embodiment of this application, the above-described construction of an affine transformation matrix using the source image and image samples may include the following steps:

[0106] Obtain the first coordinate information of preset feature points of the target protected object in the source image;

[0107] Obtain the second coordinate information of preset feature points of sample objects in each image sample;

[0108] The affine transformation matrix is ​​calculated using the first coordinate information and each of the second coordinate information.

[0109] This application provides an implementation method for constructing an affine transformation matrix, which is based on the coordinate information of the source image and the coordinate information of the image samples. First, the target protected object in the source image is determined, and the coordinate information of preset feature points in the target protected object is obtained, i.e., the first coordinate information mentioned above. Further, the sample objects in each sample image are determined, and the coordinate information of preset feature points in each sample object is obtained, i.e., the second coordinate information mentioned above. Finally, the first coordinate information and each second coordinate information are combined to construct the final affine transformation matrix.

[0110] The preset feature points are feature points pre-defined in the image object. Taking a face image as an example, the preset feature points can be the center point of the left eyeball, the center point of the right eyeball, the tip of the nose, the left corner of the mouth, and the right corner of the mouth. Naturally, the more preset feature points, the higher the accuracy of the constructed affine transformation matrix. Experiments show that at least three feature points should be included when constructing an affine transformation matrix.

[0111] In one embodiment of this application, the above-mentioned calculation of the affine transformation matrix using the first coordinate information and each of the second coordinate information may include: using the least squares estimation algorithm to calculate the first coordinate information and each of the second coordinate information to obtain the affine transformation matrix.

[0112] This application provides a method for constructing an affine transformation matrix based on coordinate information, which can be implemented using the least squares estimation algorithm. The implementation principle of the least squares estimation algorithm is based on existing technologies and will not be elaborated further here.

[0113] This application provides another image protection method.

[0114] This application uses a face image as an example to introduce an image protection method, the implementation process of which may include the following steps:

[0115] Step 1: Input the source face image, the target image (i.e., the random image mentioned above), and the face recognition model:

[0116] (1) Face source image:

[0117] The source image for a face is typically a three-channel color face image (R channel, G channel, and B channel), a single-channel grayscale face image, or other face image formats. The following description uses a three-channel color face image as an example. Of course, it can also be easily extended to accommodate single-channel grayscale face images or other face image formats.

[0118] The source image of a face can be a face image captured by any shooting device, such as a camera, mobile phone, or surveillance camera. Typically, the face images captured by these devices are high-resolution, three-channel color face images, for example, face images contained in a 1920*1080 pixel color image.

[0119] Furthermore, a standardized face source image (i.e., a face image that conforms to the input image specifications of the face recognition model, such as a 112*112 pixel upright face image) is obtained from the face source image. The usual implementation method is to use image transformation methods such as rotation, scaling or affine transformation to transform the face region on the face source image into a standardized face source image.

[0120] In a preferred implementation:

[0121] S1: Obtain the two-dimensional coordinates of five feature points (center of the left eyeball, center of the right eyeball, tip of the nose, left corner of the mouth, and right corner of the mouth) of a 112*112 pixel upright "average" face. First, collect a large number of face images and manually label the two-dimensional coordinates of the feature points. Then, use mathematical statistics methods to calculate the two-dimensional coordinates of the five feature points of a 112*112 pixel upright average face, defined as {x... i ,y i}, i = 1, 2, 3, 4, 5. The 112*112 pixel value is only used as an example.

[0122] S2: Obtain the two-dimensional coordinates of five feature points of the face to be protected on the source face image. Mark the two-dimensional coordinates of the five feature points on the source face image and define them as {x′}. i ,y′ i}, i = 1, 2, 3, 4, 5. The annotation method can be either manual annotation of coordinates or automatic acquisition of the coordinates of the five feature points using a high-precision face feature point localization algorithm.

[0123] S3: Using the two sets of coordinates mentioned above, i.e. {x} i ,y i}, i = 1, 2, 3, 4, 5 and {x′ i ,y′ i}, i=1,2,3,4,5, we can calculate {x′ i ,y′ i}, i = 1, 2, 3, 4, 5 to {x i ,y i The affine transformation matrix M, where i = 1, 2, 3, 4, 5, can be implemented using the least squares estimation algorithm. Then, the affine transformation matrix M is used to transform the face region on the face source image into a 112*112 pixel face image, which is the standardized face source image.

[0124] When calculating the affine transformation matrix M, at least three feature points should be included, such as the center of the left eyeball, the center of the right eyeball, and the tip of the nose. Generally, using more feature points results in higher accuracy of the affine transformation matrix M; the above five feature points are a preferred combination.

[0125] (2) Target image:

[0126] The purpose of this technical solution is to generate a protected face image (which, if illegally obtained by an attacker, cannot be used to attack a deep learning-based face recognition system). From a human visual perspective, the protected face image and the standardized face source image represent the same person; that is, a human observer perceives the protected face image and the standardized face source image as belonging to the same person. Furthermore, in the feature space of the face recognition model, the distance between the features of the protected face image and the features of the target image is smaller than the distance between the features of the protected face image and the features of the standardized face source image.

[0127] Based on the above theoretical requirements, the target image should be a face image of another person, an image without a face, or an automatically generated random image. In this embodiment, the target image is assumed to be a face image of another person, and a standardized face source image generation method can be applied. The target image is also a 112*112 pixel face image (a face image conforming to the input image specifications of the face recognition model).

[0128] (3) Face recognition model:

[0129] The face recognition model uses ResNet50 as the backbone network. This face recognition model is trained on a large-scale face recognition training set using the stochastic gradient descent method. The loss function used during training is ArcFace loss.

[0130] The detailed configuration of the face recognition model is as follows: ResNet50 includes 5 convolutional layers conv1, conv2_x, conv3_x, conv4_x, and conv5_x, as well as BN (Batch Normalization), dropout, and FC (Fully-connected layer). The conv1 layer is a convolutional layer with a stride of 1. The input to the FC layer is the output feature map of the conv5_x layer passed through the BN and dropout layers sequentially. The output of the FC layer is a 512-dimensional feature (in this technical solution, the 512-dimensional feature output by the face recognition model is also referred to as face feature, face feature vector, or feature vector).

[0131] Step 2: Initialize the protected face image:

[0132] The protected face image (denoted as f) is generated using a standardized face source image (denoted as s), a target image (denoted as t), and a face recognition model (denoted as Φ(*)) through an iterative optimization algorithm. Furthermore, the protected face image is initialized using the standardized face source image: f = s.

[0133] From a visual perspective, the final generated protected face image is similar to the standardized face source image. Therefore, the above initialization method can initialize the protected face image near the target value, which can accelerate the convergence of the algorithm. However, using other methods (such as random initialization) to initialize the protected face image may cause the algorithm to diverge.

[0134] Step 3: Calculate the feature distance between the protected face image (f) and the target image (t):

[0135] S1: Input the face image to be protected into the face recognition model Φ(*) (f), and after forward propagation of the neural network, obtain the feature vector V of the face image (f) to be protected. f V f =Φ(f), according to the definition of the face recognition model, the feature vector is a 512-dimensional vector.

[0136] S2: Input the target image (t) into the face recognition model Φ(*), and after forward propagation of the neural network, obtain the feature vector V of the target image (t). t V t =Φ(t), according to the definition of the face recognition model, the feature vector is a 512-dimensional feature vector.

[0137] S3: Feature vector V f and V t The distance between them is represented by cosine distance. The eigenvector V is calculated according to the following formula. f and V t Cosine distance between them:

[0138]

[0139] Where · represents the dot product operation, and ||*||2 represents the L2 norm.

[0140] Step 4: Calculate the feature distance between the protected face image (f) and the normalized source face image (s):

[0141] S1: Input the standardized face source image (s) into the face recognition model Φ(*), and after forward propagation of the neural network, obtain the feature vector V of the standardized face source image (s). s V s =Φ(s), according to the definition of the face recognition model, the feature vector is a 512-dimensional feature vector.

[0142] S2: Feature vector V f and V s The distance between them is represented by cosine distance. The eigenvector V is calculated according to the following formula. fand V s Cosine distance between them:

[0143]

[0144] Where · represents the dot product operation, and ||*||2 represents the L2 norm.

[0145] Step 5: Calculate the apparent distance between the protected face image (f) and the normalized face source image (s):

[0146] To ensure that the final generated protected face image (f) is as visually similar as possible to the standardized face source image (s), the output of the intermediate layer (ReLU2-2 layer) of the VGG-16 image classification network can be used as a representation of the image appearance to measure the appearance distance. The VGG-16 network is an image classification network pre-trained on the ImageNet dataset.

[0147]

[0148] Where F(*) represents the output feature vector of the intermediate layer (ReLU2-2 layer) extracted using the VGG-16 image classification network.

[0149] Alternatively, the apparent distance can be calculated using the F-norm of the protected face image (f) and the normalized face source image (s). Then, D vis =||fs|| F , where ||*|| F This represents the F-norm.

[0150] Step 6: Calculate the overall loss function:

[0151] The formula for calculating the loss function is:

[0152] L(f,s,t)=max(T f→t -D f→t ,0)+λ1 max(D f→s -T f→s ,0)+λ2D vis ;

[0153] L(f,s,t) represents the loss value;

[0154] D f→t D represents the distance between the feature vectors of the protected face image (f) and the target image (t); f→s D represents the distance between the feature vector of the protected face image (f) and the feature vector of the normalized source face image (s); vis The apparent distance between the protected face image (f) and the normalized face source image (s) is represented.

[0155] T f→t It is a threshold (set T) f→t ≥0.5), used to constrain D f→t When D f→t ≥T f→t At that time, T f→t -D f→t ≤0, max(T) f→t -D f→t ,0)=0,max(T f→t -D f→t The meaning of ,0) is: when D f→t >T f→t When the feature vectors of image (f) and image (t) are sufficiently close in the feature space, the loss value is zero.

[0156] T f→s It is a threshold (set T) f→s ≤0.5), used to constrain D f→s When D f→s ≤T f→s At that time, D f→s -T f→s ≤0, max(D f→s -T f→s ,0)=0,max(D f→s -T f→s The meaning of ,0) is: when D f→s <T f→s When the feature vectors of image (f) and image (s) are far enough apart in the feature space, the loss value is zero.

[0157] max(*,*) means taking the maximum of the two inputs;

[0158] The coefficient λ1 is used to balance the importance of the first two terms, and λ1 = 1.0 is taken.

[0159] The λ² coefficient is used to constrain the apparent distance D. vis Let λ2 = 1 / L, where L represents the length of the output feature vector of the intermediate layer (ReLU2-2 layer) extracted by the VGG-16 image classification network; if the apparent distance is calculated using the F norm between image matrices, let λ2 = 1 / (3*w*h), where w represents the image width, h represents the image height, and "3" represents the number of image channels.

[0160] The first term of the loss function calculation formula is used to push the protected face image (f) closer to the target image (t) in the feature space; the second term of the loss function calculation formula is used to push the protected face image (f) away from the standardized face source image (s) in the feature space; and the third term of the loss function calculation formula is used to ensure that the protected face image (f) is visually close to the face source image (s).

[0161] Step 7: Update the protected face image (f) using the backpropagation algorithm:

[0162] Assuming the initial protected face image is f0 and the face source image is (s), then f0 = s.

[0163] The rule for updating the protected face image (f) is as follows:

[0164]

[0165] Where j = 1, 2, ..., j_max_iters, j represents the j-th iteration update, j_max_iters represents the maximum number of iterations, j_max_iters = 1000; lr represents the learning rate, lr = 0.01; Represents finding L(f) j-1 ,s,t) for f j-1 The derivative of .

[0166] After the iteration is complete, f j_max_iters This is the final protected image of the face. During training, the parameters of the face recognition model remain constant; the parameters of the VGG-16 image classification network used to extract visual features also remain constant.

[0167] As can be seen, the image protection method provided in this application takes into account both the feature distance between the protected image and the source image and the feature distance between the protected image and the random image. Based on the face recognition model and the backpropagation algorithm, it compresses the feature distance between the protected image and the random image and expands the feature distance between the protected image and the source image. This makes the final generated protected image closer to the random image and further away from the source image. In other words, the final generated protected image containing the target object is closer to the random image that does not contain the target object and further away from the original image containing the target object. At this time, the target object recognition system will have difficulty identifying the target object in the protected image, thereby achieving privacy protection of the target object. This can effectively prevent the image from being forged or stolen for illegal attack purposes and ensure information security.

[0168] This application provides an image protection device.

[0169] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of an image protection device provided in this application. The image protection device may include:

[0170] Module 1 is used to acquire source images and random images. The source image includes the target protected object, while the random image does not include the target protected object.

[0171] Processing module 2 is used to generate an initial protected image based on the source image.

[0172] The first calculation module 3 is used to calculate the first feature distance between the initial protected image and the random image, the second feature distance between the initial protected image and the source image, and the apparent distance;

[0173] The second calculation module 4 is used to calculate the loss function based on the first feature distance, the second feature distance, and the apparent distance.

[0174] Update module 5 is used to iteratively update the initial protected image based on the loss function and using the backpropagation algorithm to obtain a protected image of the target protected object.

[0175] As can be seen, the image protection device provided in this application takes into account both the feature distance between the protected image and the source image and the feature distance between the protected image and the random image. Based on the face recognition model and the backpropagation algorithm, it compresses the feature distance between the protected image and the random image and expands the feature distance between the protected image and the source image, so that the final generated protected image is far away from the source image and close to the random image. In other words, the final generated protected image containing the target object is closer to the random image that does not contain the target object and far away from the original image containing the target object. At this time, the target object recognition system will have difficulty identifying the target object in the protected image, thereby realizing the privacy protection of the target object. It can effectively prevent the image from being forged or stolen for illegal attack purposes and ensure information security.

[0176] In one embodiment of this application, the first calculation module 3 can be specifically used to process the initial protected image and the random image using a face recognition model to obtain a first feature vector and a second feature vector; and to calculate the cosine distance between the first feature vector and the second feature vector to obtain the first feature distance.

[0177] In one embodiment of this application, the first calculation module 3 described above can be specifically used to process the source image using a face recognition model to obtain a third feature vector; and to calculate the cosine distance between the first feature vector and the third feature vector to obtain a second feature distance.

[0178] In one embodiment of this application, the face recognition model described above can be a neural network model based on ResNet50.

[0179] In one embodiment of this application, the first calculation module 3 described above can be specifically used to perform F-norm calculation on the initial protected image and the source image to obtain the apparent distance.

[0180] In one embodiment of this application, the first calculation module 3 described above can be specifically used to input the initial protected image and the source image into an image classification network for processing to obtain the apparent distance.

[0181] In one embodiment of this application, the image classification network described above can be a VGG-16-based neural network model.

[0182] In one embodiment of this application, the image protection device may further include a standardization module, which is used to standardize the source image and the random image according to the input specifications of the face recognition model before generating the initial protection image based on the source image, to obtain a standardized source image and a standardized random image; wherein the image format of the standardized source image and the standardized random image is the image format specified by the face recognition model.

[0183] In one embodiment of this application, the standardization module described above may include:

[0184] The recognition unit is used to construct an affine transformation matrix using the source image and image samples;

[0185] The processing module is used to convert the source image into a normalized source image using an affine transformation matrix.

[0186] In one embodiment of this application, the standardization module described above may include:

[0187] The building unit is used to construct an affine transformation matrix using the source image and image samples;

[0188] The transformation unit is used to convert a source image into a normalized source image using an affine transformation matrix.

[0189] In one embodiment of this application, the above-mentioned building unit may include:

[0190] The first acquisition subunit is used to acquire the first coordinate information of the preset feature points of the target protected object in the source image;

[0191] The second acquisition subunit is used to acquire the second coordinate information of the preset feature points of the sample objects in each image sample;

[0192] The computational sub-unit is used to calculate the affine transformation matrix using the first coordinate information and each of the second coordinate information.

[0193] In one embodiment of this application, the above-mentioned calculation subunit can be specifically used to calculate the first coordinate information and each of the second coordinate information using the least squares estimation algorithm to obtain the affine transformation matrix.

[0194] For a description of the apparatus provided in the embodiments of this application, please refer to the above method embodiments; further details will not be repeated here.

[0195] This application provides an image protection device.

[0196] Please refer to Figure 3 , Figure 3 This application provides a schematic diagram of the structure of an image protection device, which may include:

[0197] Memory, used to store computer programs;

[0198] A processor, used to execute computer programs, can implement the steps of any of the image protection methods described above.

[0199] like Figure 3 The diagram shows the structural composition of an image protection device, which may include a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 communicate with each other via the communication bus 13.

[0200] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0201] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in the embodiments of the image protection method.

[0202] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0203] Acquire a source image and a random image. The source image includes the target protected object, while the random image does not.

[0204] Initialize and generate an initial protected image based on the source image;

[0205] Calculate the first feature distance between the initial protected image and the random image, the second feature distance between the initial protected image and the source image, and the apparent distance;

[0206] The loss function is calculated based on the first feature distance, the second feature distance, and the apparent distance.

[0207] Based on the loss function, the backpropagation algorithm is used to iteratively update the initial protected image to obtain a protected image of the target protected object.

[0208] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0209] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0210] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0211] Of course, it should be noted that, Figure 3 The structure shown does not constitute a limitation on the image protection device in the embodiments of this application. In practical applications, the image protection device may include more than Figure 3 More or fewer components as shown, or combinations of certain components.

[0212] This application provides a computer-readable storage medium.

[0213] The computer-readable storage medium provided in this application embodiment stores a computer program, which, when executed by a processor, can implement the steps of any of the image protection methods described above.

[0214] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0215] For a description of the computer-readable storage medium provided in the embodiments of this application, please refer to the above method embodiments; further details will not be repeated here.

[0216] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0217] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0218] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0219] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. An image protection method, characterized in that, The method includes: Acquire a source image and a random image, wherein the source image includes the target protected object and the random image does not include the target protected object; An initial protection image is generated based on the source image; Calculate the first feature distance between the initial protected image and the random image, the second feature distance between the initial protected image and the source image, and the apparent distance; The loss function is calculated based on the first feature distance, the second feature distance, and the apparent distance. Based on the loss function, the initial protected image is iteratively updated using the backpropagation algorithm to obtain a protected image of the target protected object. Calculating the apparent distance between the initial protected image and the source image includes: performing F-norm calculation on the initial protected image and the source image to obtain the apparent distance; or, inputting the initial protected image and the source image into an image classification network for processing to obtain the apparent distance.

2. The method according to claim 1, characterized in that, Calculating the first feature distance between the initial protected image and the random image includes: The initial protected image and the random image are processed using a face recognition model to obtain a first feature vector and a second feature vector. Calculate the cosine distance between the first feature vector and the second feature vector to obtain the first feature distance.

3. The method according to claim 2, characterized in that, Calculating the second feature distance between the initial protected image and the source image includes: The source image is processed using the face recognition model to obtain a third feature vector; Calculate the cosine distance between the first feature vector and the third feature vector to obtain the second feature distance.

4. The method according to claim 3, characterized in that, The face recognition model is a neural network model based on ResNet50.

5. The method according to claim 1, characterized in that, The image classification network is a neural network model based on VGG-16.

6. The method according to any one of claims 2 to 5, characterized in that, Before generating the initial protected image based on the source image, the process further includes: Based on the input specifications of the face recognition model, the source image and the random image are standardized respectively to obtain standardized source image and standardized random image; The standardized source image and the standardized random image are in the image format specified by the face recognition model.

7. The method according to claim 6, characterized in that, The source image is standardized to obtain the standardized source image, including: Perform type recognition on the source image to determine the image type; The source image is standardized by using the processing strategy corresponding to the image type to obtain the standardized source image.

8. The method according to claim 6, characterized in that, The source image is standardized to obtain the standardized source image, including: Construct an affine transformation matrix using the source image and image samples; The source image is converted into the normalized source image using the affine transformation matrix.

9. The method according to claim 8, characterized in that, The step of constructing an affine transformation matrix using the source image and image samples includes: Obtain the first coordinate information of the preset feature points of the target protected object in the source image; Obtain the second coordinate information of preset feature points of sample objects in each of the image samples; The affine transformation matrix is ​​calculated using the first coordinate information and each of the second coordinate information.

10. The method according to claim 9, characterized in that, The step of calculating the affine transformation matrix using the first coordinate information and each of the second coordinate information includes: The affine transformation matrix is ​​obtained by calculating the first coordinate information and each of the second coordinate information using the least squares estimation algorithm.

11. An image protection device, characterized in that, The device includes: The acquisition module is used to acquire a source image and a random image, wherein the source image includes the target protected object and the random image does not include the target protected object; The processing module is used to initialize and generate an initial protection image based on the source image; The first calculation module is used to calculate the first feature distance between the initial protected image and the random image, and the second feature distance and apparent distance between the initial protected image and the source image; The second calculation module is used to calculate a loss function based on the first feature distance, the second feature distance, and the apparent distance. The update module is used to iteratively update the initial protected image based on the loss function using a backpropagation algorithm to obtain a protected image of the target protected object. The first calculation module is specifically used to perform F-norm calculation on the initial protected image and the source image to obtain the apparent distance; or, input the initial protected image and the source image into an image classification network for processing to obtain the apparent distance.

12. An image protection device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the image protection method as described in any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image protection method as described in any one of claims 1 to 10.

Images