Image protection method, device, electronic device and storage medium
By selecting attack images with high similarity to the face images to be processed and generating post-processed face images with disturbed information, the problem of personal information leakage caused by face recognition technology is solved, and the attacks are achieved are anti-aggressive and user privacy is protected.
Patent Information
- Application Number
- CN202011007822.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-23
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-09-23
AI Technical Summary
The widespread application of deep learning-based facial recognition technology may lead to the leakage of personal information, especially when a user publishes images containing facial information, other platforms or individuals can obtain user's personal information through these images.
By obtaining the similarity between the to-processed face image and multiple candidate attack images, selecting the target attack image, and processing the to-processed face image based on the target attack image, generating a processed face image containing perturbation information. It maintains indiscriminateness at the user's visual level, while making it impossible for illegal stealers to obtain real information about faces, achieving the purpose of fighting attacks.
It effectively protects users' personal information, prevents the abuse of facial recognition technology, ensures that the facial images are visually no different from the original image after processing, and prevents illegal acquisition of real facial information.
Smart Images

Figure CN112149732B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of deep learning technology, and in particular to an image protection method, device, electronic device and storage medium. Background Art
[0002] With the rapid development of deep learning, face recognition technology based on deep learning has been widely used in various scenarios, such as face attendance, face payment, etc. Specifically, face recognition technology based on deep learning extracts features from face images and determines the personal identity information of face images by feature comparison.
[0003] The widespread use of facial recognition technology may lead to the leakage of personal information. For example, when a user posts an image containing facial information on a public platform, the image containing facial information may be uploaded to the Internet, allowing other platforms or other individuals to obtain the user's personal information through the image containing facial information, resulting in the leakage of personal information. Summary of the invention
[0004] In view of this, the present disclosure at least provides an image protection method, device, electronic device and storage medium.
[0005] In a first aspect, the present disclosure provides an image protection method, comprising:
[0006] Obtain the face image to be processed;
[0007] Acquire the similarity between the face image to be processed and a plurality of candidate attack images, and select a target attack image from the candidate attack images based on the similarity;
[0008] The face image to be processed is processed based on the target attack image to generate a processed face image containing disturbance information; the disturbance information is used to interfere with the detection of real face information in the face image to be processed.
[0009] In the above implementation, a target attack image is selected from the candidate attack images based on the similarity between the candidate attack images and the face image to be processed. For example, the target attack image may be a face image with a high similarity to the face image to be processed. In this way, a processed face image containing disturbance information is generated based on the target attack image and the face image to be processed. On the one hand, there is basically no difference between the processed face image and the face image to be processed at the user visual level. On the other hand, the addition of disturbance information can prevent the party that illegally steals user information from obtaining the real face information corresponding to the face image to be processed, thereby achieving the purpose of countering attacks.
[0010] In a possible implementation, selecting a target attack image from the candidate attack images based on the similarity includes:
[0011] From the multiple candidate attack images, the candidate attack image having the highest similarity to the face image to be processed is selected as the target attack image.
[0012] In a possible implementation manner, obtaining the similarity between the to-be-processed face image and a plurality of candidate attack images includes:
[0013] Extracting facial features of the facial image to be processed;
[0014] Calculate the feature similarity between the facial features of the face image to be processed and the facial features corresponding to each of the multiple candidate attack images, and use the feature similarity as the similarity between the face image to be processed and the candidate attack images.
[0015] In a possible implementation manner, the plurality of candidate attack images have a plurality of different facial attributes;
[0016] The facial attributes include, but are not limited to: skin color, gender, face orientation, face shape, and facial features.
[0017] Here, the stored multiple candidate attack images have different facial attributes. By setting different facial attributes, a target attack image that is relatively similar to the facial image to be processed can be found.
[0018] In a possible implementation, the processing of the to-be-processed face image based on the target attack image to generate a processed face image containing disturbance information includes:
[0019] Performing feature extraction on the target attack image to generate a first feature map corresponding to the target attack image, and performing feature extraction on the face image to be processed to generate a second feature map corresponding to the face image to be processed;
[0020] Based on the first feature map and the second feature map, a disturbance image is generated; the disturbance image contains disturbance information;
[0021] Based on the disturbance image and the face image to be processed, the processed face image is generated.
[0022] In the above implementation, a disturbed image is generated by extracting the first feature map and the second feature map, and the disturbed image contains disturbance information. Then, based on the disturbed image and the face image to be processed, a processed face image is generated.
[0023] In a possible implementation, generating a perturbed image based on the first feature map and the second feature map includes:
[0024] Extracting difference features from a combined feature map of the first feature map and the second feature map to obtain a difference feature map representing difference information between the first feature map and the second feature map;
[0025] Based on the difference feature map, the perturbed image is generated.
[0026] In the above implementation, a difference feature map is obtained by performing difference feature extraction on the combined feature map of the first feature map and the second feature map. The difference feature map can represent the difference information between the first feature map and the second feature map, so that the generated disturbance image contains the difference information. After the processed face image is generated based on the disturbance image and the face image to be processed, on the one hand, there is basically no difference between the processed face image and the face image to be processed at the user visual level. On the other hand, the facial feature information of the processed face image includes facial feature information of the target attack image that is different from the facial feature information of the face image to be processed. As a result, when the processed face image is detected, the facial feature information corresponding to the face image to be processed cannot be obtained, thereby achieving the purpose of countering attacks.
[0027] In a possible implementation, generating the processed facial image based on the disturbed image and the facial image to be processed includes:
[0028] Performing visible pixel weakening processing on pixel information of pixel points on the disturbed image to generate a processed disturbed image;
[0029] Based on the processed disturbance image and the face image to be processed, the processed face image is generated.
[0030] Here, in order to reduce the influence of the visual features on the disturbed image on the face image to be processed, the pixel information of the pixels on the disturbed image can be firstly weakened to generate a processed disturbed image, and then based on the processed disturbed image and the face image to be processed, a processed face image is generated, so as to achieve the purpose of countering the attack while ensuring the display effect of the face image.
[0031] In a possible implementation manner, the processed face image containing disturbance information is generated based on a trained target neural network; the trained target neural network is obtained by the following steps:
[0032] Obtain face image samples;
[0033] Based on the face image sample and the neural network to be trained, determining an attack image sample corresponding to the face image sample;
[0034] Based on the attack image sample, the face image sample, and the neural network to be trained, generating a processed face image sample;
[0035] Based on the attack image samples and the processed face image samples, the neural network to be trained is trained, and / or based on the face image samples and the processed face image samples, the neural network to be trained is trained to obtain a trained target neural network.
[0036] In a possible implementation, training the neural network to be trained based on the attack image sample and the processed face image sample includes:
[0037] Extracting a first image feature corresponding to the processed face image sample and a second image feature corresponding to the attack image sample;
[0038] Calculating a first loss value of the neural network to be trained based on the first image feature and the second image feature;
[0039] The neural network to be trained is trained using the first loss value.
[0040] In the above implementation, a first loss value is generated and the neural network to be trained is trained using the first loss value to obtain a trained target neural network. A processed face image is generated based on the trained target neural network. This allows the facial feature information corresponding to the extracted processed face image to have the facial feature information of the target attack image, thereby achieving an effect of countering the attack.
[0041] In a possible implementation, training the neural network to be trained based on the face image sample and the processed face image sample includes:
[0042] Extracting a first image feature corresponding to the processed face image sample and a third image feature corresponding to the face image sample;
[0043] Calculating a second loss value of the neural network to be trained based on the first image feature and the third image feature;
[0044] The neural network to be trained is trained using the second loss value.
[0045] In the above implementation, by generating a second loss value, the neural network to be trained is trained using the second loss value to obtain a trained target neural network, and a processed facial image is generated based on the trained target neural network. This allows the facial feature information corresponding to the extracted processed facial image to be distinguished from the facial feature information of the facial image to be processed, thereby achieving an effect of counteracting attacks.
[0046] The effects of the following devices, electronic devices, etc. are described in the description of the above method and will not be repeated here.
[0047] In a second aspect, the present disclosure provides an image protection device, comprising:
[0048] An acquisition module, used for acquiring a face image to be processed;
[0049] A selection module, configured to obtain similarities between the face image to be processed and a plurality of candidate attack images, and select a target attack image from the candidate attack images based on the similarities;
[0050] A generation module is used to process the face image to be processed based on the target attack image to generate a processed face image containing disturbance information; the disturbance information is used to interfere with the detection of real face information in the face image to be processed.
[0051] In a possible implementation manner, the selection module, when selecting a target attack image from the candidate attack images based on the similarity, is configured to:
[0052] From the multiple candidate attack images, the candidate attack image having the highest similarity to the face image to be processed is selected as the target attack image.
[0053] In a possible implementation, when obtaining the similarity between the face image to be processed and a plurality of candidate attack images, the selection module is used to:
[0054] Extracting facial features of the facial image to be processed;
[0055] Calculate the feature similarity between the facial features of the face image to be processed and the facial features corresponding to each of the multiple candidate attack images, and use the feature similarity as the similarity between the face image to be processed and the candidate attack images.
[0056] In a possible implementation manner, the plurality of candidate attack images have a plurality of different facial attributes;
[0057] The facial attributes include, but are not limited to: skin color, gender, face orientation, face shape, and facial features.
[0058] In a possible implementation manner, the generating module, when processing the to-be-processed face image based on the target attack image to generate a processed face image containing disturbance information, is configured to:
[0059] Performing feature extraction on the target attack image to generate a first feature map corresponding to the target attack image, and performing feature extraction on the face image to be processed to generate a second feature map corresponding to the face image to be processed;
[0060] Based on the first feature map and the second feature map, a disturbance image is generated; the disturbance image contains disturbance information;
[0061] Based on the disturbance image and the face image to be processed, the processed face image is generated.
[0062] In a possible implementation, when the generating module generates the perturbed image based on the first feature map and the second feature map, it is configured to:
[0063] Extracting difference features from a combined feature map of the first feature map and the second feature map to obtain a difference feature map representing difference information between the first feature map and the second feature map;
[0064] Based on the difference feature map, the perturbed image is generated.
[0065] In a possible implementation manner, when the generating module generates the processed facial image based on the disturbed image and the facial image to be processed, it is configured to:
[0066] Performing visible pixel weakening processing on pixel information of pixel points on the disturbed image to generate a processed disturbed image;
[0067] Based on the processed disturbance image and the face image to be processed, the processed face image is generated.
[0068] In a possible implementation manner, the processed face image containing disturbance information is generated based on a trained target neural network; the device further comprises: a training module, configured to obtain the trained target neural network through the following steps:
[0069] Obtain face image samples;
[0070] Based on the face image sample and the neural network to be trained, determining an attack image sample corresponding to the face image sample;
[0071] Based on the attack image sample, the face image sample, and the neural network to be trained, generating a processed face image sample;
[0072] Based on the attack image samples and the processed face image samples, the neural network to be trained is trained, and / or based on the face image samples and the processed face image samples, the neural network to be trained is trained to obtain a trained target neural network.
[0073] In a possible implementation manner, the training module, when training the neural network to be trained based on the attack image sample and the processed face image sample, is used to:
[0074] Extracting a first image feature corresponding to the processed face image sample and a second image feature corresponding to the attack image sample;
[0075] Calculating a first loss value of the neural network to be trained based on the first image feature and the second image feature;
[0076] The neural network to be trained is trained using the first loss value.
[0077] In a possible implementation manner, the training module, when training the neural network to be trained based on the face image samples and the processed face image samples, is used to:
[0078] Extracting a first image feature corresponding to the processed face image sample and a third image feature corresponding to the face image sample;
[0079] Calculating a second loss value of the neural network to be trained based on the first image feature and the third image feature;
[0080] The neural network to be trained is trained using the second loss value.
[0081] In a third aspect, the present disclosure provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image protection method described in the first aspect or any one of the embodiments are performed.
[0082] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the image protection method as described in the first aspect or any one of the embodiments are executed.
[0083] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0084] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0085] Figure 1 A schematic diagram of a process of an image protection method provided by an embodiment of the present disclosure is shown;
[0086] Figure 2 A schematic diagram of the architecture of an image protection device provided by an embodiment of the present disclosure is shown;
[0087] Figure 3 A schematic structural diagram of an electronic device 300 provided in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0088] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0089] Considering the widespread application of face recognition technology, personal information leakage may occur. For example, after a user posts an image containing face information on a public platform, the image containing face information may be uploaded to the Internet, allowing other platforms or other individuals to obtain the user's personal information through the image containing face information, resulting in personal information leakage. Therefore, in order to solve the above problem, the embodiment of the present disclosure provides an image protection method.
[0090] To facilitate understanding of the embodiments of the present disclosure, an image protection method disclosed in the embodiments of the present disclosure is first introduced in detail. The execution subject of the image protection method provided in the embodiments of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.
[0091] See also Figure 1 FIG. 1 is a flow chart of an image protection method provided by an embodiment of the present disclosure, the method comprising S101-S103, wherein:
[0092] S101, obtaining a face image to be processed.
[0093] S102, obtaining similarities between the face image to be processed and a plurality of candidate attack images, and selecting a target attack image from the candidate attack images based on the similarities.
[0094] S103, processing the face image to be processed based on the target attack image to generate a processed face image containing disturbance information; the disturbance information is used to interfere with the detection of real face information in the face image to be processed.
[0095] In the above method, based on the similarity between the candidate attack images and the face image to be processed, a target attack image is selected from the candidate attack images. For example, the target attack image can be a face image with a high similarity to the face image to be processed. In this way, based on the target attack image and the face image to be processed, a processed face image containing disturbance information is generated. On the one hand, there is basically no difference between the processed face image and the face image to be processed at the user visual level. On the other hand, the addition of disturbance information can prevent the party that illegally steals user information from obtaining the real face information corresponding to the face image to be processed, thereby achieving the purpose of countering attacks.
[0096] For S101 and S102:
[0097] Here, the face image to be processed may be any image containing face information posted by a user, and the image may be an image in a red-green-blue (RGB) format.
[0098] In an optional implementation, the plurality of candidate attack images have a plurality of different facial attributes; wherein the facial attributes include but are not limited to: skin color, gender, face orientation, face shape, and facial features.
[0099] Here, the stored multiple candidate attack images have different facial attributes. By setting multiple different facial attributes, a target attack image that is relatively similar to the facial image to be processed can be found.
[0100] For example, skin color can include yellow, black, and white; gender can be divided into male and female; facial orientation can be divided into forward, left, and right; face shape can be divided into square face, inverted triangle face, long face, round face, triangular face, and diamond face; facial features include eye shape, mouth shape, eyebrow shape, nose shape, and ear shape; eye shape can include: peach blossom eyes, phoenix eyes, almond eyes, etc.; mouth shape can include: thick lips, thin lips, etc.; eyebrow shape can include: straight eyebrows, willow leaf eyebrows, etc.; nose shape can include: hooked nose, sword nose, etc. Among them, the facial attributes of the candidate attack image can be set according to actual needs, which is only an exemplary description here. For example, facial attributes can also include whether to wear glasses; the shape of the forehead, etc.
[0101] Multiple candidate attack images with different facial attributes can be acquired in advance, and the number of candidate attack images can be determined according to actual needs. For example, 100 candidate attack images with different facial attributes can be acquired. Exemplarily, when the facial attributes include skin color and gender, the skin color includes yellow, black, and white, and the gender includes male and female, so a candidate attack image with yellow skin color and male gender, a candidate attack image with yellow skin color and female gender, a candidate attack image with black skin color and male gender, a candidate attack image with black skin color and female gender, a candidate attack image with white skin color and male gender, and a candidate attack image with white skin color and female gender can be acquired, and the acquired multiple candidate attack images are stored.
[0102] Here, it is proved through multiple experiments that when a candidate attack image with a high similarity to the face image to be processed is selected as the attack target, the success rate of the adversarial attack task can be higher. That is, when a candidate attack image with a high similarity to the face image to be processed is selected as the target attack image, a processed face image containing perturbation information is generated. When other platforms or other individuals recognize the processed face image, they can obtain the face information of the person in the target attack image, but cannot obtain the real face information contained in the face image to be processed, thereby realizing the protection of the face image to be processed and achieving the purpose of adversarial attack.
[0103] Therefore, after obtaining the face image to be processed, the similarity between each candidate attack image in the plurality of candidate attack images and the face image to be processed can be calculated respectively, and based on the similarity corresponding to each candidate attack image, a target attack image can be selected from the plurality of candidate attack images. That is, the candidate attack image with the highest similarity can be selected as the target attack image.
[0104] Among them, selecting a target attack image from the candidate attack images based on the similarity may include: selecting, from the multiple candidate attack images, a candidate attack image with the highest similarity to the face image to be processed as the target attack image.
[0105] In an optional implementation manner, obtaining the similarity between the face image to be processed and a plurality of candidate attack images includes:
[0106] S1021, extracting facial features of the facial image to be processed.
[0107] S1022, calculating feature similarity between facial features of the face image to be processed and facial features corresponding to each of the multiple candidate attack images, and using the feature similarity as similarity between the face image to be processed and the candidate attack images.
[0108] Here, the facial features of the facial image to be processed can be extracted by the facial feature extraction module, and the facial features can be a feature set corresponding to the face, for example, the facial features can be round face, female, yellow skin and other features; or, the facial features can also be a feature map corresponding to the facial image to be processed, and the feature map contains facial feature information.
[0109] And the facial features of each candidate attack image in the multiple candidate attack images can also be extracted by the facial feature extraction module. In one embodiment, after obtaining the multiple candidate attack images, the facial features of each candidate attack image can be extracted, and the facial features of each candidate attack image can be associated with the corresponding candidate attack image and stored, so that after the facial features of the face image to be processed are extracted, the feature similarity between the facial features of the face image to be processed and the facial features corresponding to each stored candidate attack image can be calculated. In another embodiment, after the facial features of the face image to be processed are extracted, the facial features of each candidate attack image can be extracted by the facial feature extraction module, and then the feature similarity between the facial features of the face image to be processed and the facial features corresponding to each stored candidate attack image can be calculated.
[0110] For example, the cosine similarity between the facial features of the face image to be processed and the facial features corresponding to each candidate attack image can be calculated, and the cosine similarity can be used as the feature similarity, that is, the similarity between the face image to be processed and the candidate attack image can be obtained. Alternatively, a similarity determination module can be used to determine the feature similarity between the facial features of the face image to be processed and the facial features corresponding to each candidate attack image.
[0111] In specific implementation, the trained attack target selection neural network can also be used to determine the target attack image with the highest similarity to the face image to be processed from multiple candidate attack images. The attack target selection neural network includes a face feature extraction module and a similarity determination module. The structure of the attack target selection neural network can be set according to actual needs.
[0112] The attack target selection neural network is trained according to the following steps: multiple groups of image samples are obtained, each group of image samples includes two face image samples with high similarity; the multiple image samples are input into the attack target selection neural network to be trained, and the attack target selection neural network to be trained is trained until the accuracy of the attack target selection neural network to be trained is greater than a set threshold, thereby obtaining a trained attack target selection neural network.
[0113] For S103:
[0114] After determining the target attack image corresponding to the face image to be processed, the face image to be processed can be processed based on the target attack image to generate a processed face image containing disturbance information, and the disturbance information is used to interfere with the detection of the real face information in the face image to be processed. That is, other platforms or other individuals cannot obtain the real face information of the face image to be processed when detecting the processed face image containing disturbance information, thereby protecting the personal information of the user who posted the face image to be processed.
[0115] In an optional implementation, in S103, processing the face image to be processed based on the target attack image to generate a processed face image containing disturbance information may include:
[0116] S1031, extracting features from the target attack image to generate a first feature map corresponding to the target attack image, and extracting features from the face image to be processed to generate a second feature map corresponding to the face image to be processed;
[0117] S1032, generating a disturbed image based on the first feature map and the second feature map; the disturbed image includes disturbance information;
[0118] S1033: Generate the processed facial image based on the disturbed image and the facial image to be processed.
[0119] In the above implementation, a disturbed image is generated by extracting the first feature map and the second feature map, and the disturbed image contains disturbance information. Then, based on the disturbed image and the face image to be processed, a processed face image is generated.
[0120] In S1031, the face image encoding module can be used to extract features of the target attack image to generate a first feature map corresponding to the target attack image; at the same time, the face image encoding module can be used to extract features of the face image to be processed to generate a second feature map corresponding to the face image to be processed, and the size information of the first feature map and the second feature map are consistent. The structure of the face image encoding module can be set according to actual needs, for example, the face image encoding module can be composed of multiple convolution layers.
[0121] In S1032, generating a disturbed image based on the first feature map and the second feature map may include:
[0122] 1. Perform differential feature extraction on a combined feature map of the first feature map and the second feature map to obtain a differential feature map representing differential information between the first feature map and the second feature map.
[0123] 2. Based on the difference feature map, generate the perturbation image.
[0124] Here, the first feature map and the second feature map may be cascaded to obtain a combined feature map. For example, if the size of the first feature map and the second feature map is 28×28×256, a combined feature map of size 28×28×512 is obtained. Then, the combined feature map may be subjected to differential feature extraction to obtain a differential feature map, in which the feature information of the first feature map and the second feature map is integrated, and the differential feature map may represent the differential information of the first feature map and the second feature map; wherein the size of the differential feature map may be determined as required, for example, the size of the differential feature map may be 28×28×512.
[0125] Furthermore, a perturbed image can be generated based on the difference feature map. In specific implementation, the difference feature map can be upsampled to generate a perturbed image with the same size as the face image to be processed. For example, if the size of the face image to be processed is 112×112×3, the size of the generated perturbed image is also 112×112×3. The upsampling process can be a transposed convolution process, that is, the difference feature map can be transposed convolution processed to generate a perturbed image with the same size as the face image to be processed.
[0126] In specific implementation, the face image encoding module can be used to extract the difference feature of the combined feature map of the first feature map and the second feature map to obtain the difference feature map. The face image decoding module can then be used to perform transposed convolution processing on the difference feature map to generate a perturbed image. The structure of the face image decoding module can be set according to actual needs.
[0127] In the above implementation, a difference feature map is obtained by performing difference feature extraction on the combined feature map of the first feature map and the second feature map. The difference feature map can represent the difference information between the first feature map and the second feature map, so that the generated disturbance image contains the difference information. After the processed face image is generated based on the disturbance image and the face image to be processed, on the one hand, there is basically no difference between the processed face image and the face image to be processed at the user visual level. On the other hand, the facial feature information of the processed face image includes facial feature information of the target attack image that is different from the facial feature information of the face image to be processed. As a result, when the processed face image is detected, the facial feature information corresponding to the face image to be processed cannot be obtained, thereby achieving the purpose of countering attacks.
[0128] In S1033, generating the processed facial image based on the disturbed image and the facial image to be processed may include:
[0129] 1. Perform visible pixel weakening processing on pixel information of pixel points on the disturbed image to generate a processed disturbed image.
[0130] 2. Based on the processed disturbance image and the face image to be processed, generate the processed face image.
[0131] Considering that there may be visual features visible to the human eye in the generated disturbed image, if the disturbed image is added to the image to be processed to generate the processed face image, the display effect of the generated processed face image will be poor. Therefore, in order to avoid the above situation, the pixel information of the pixel points on the disturbed image can be weakened to generate the processed disturbed image, and then the processed face image is generated based on the processed disturbed image and the face image to be processed.
[0132] Here, different l p The norm generates different constraints and performs visible pixel weakening processing on the pixel information of each pixel point on the perturbed image. Where p can be 1, 2, or infinity.
[0133] When p is 2, the pixel information of each pixel on the disturbed image can be subjected to visible pixel weakening processing according to the following formula (1) to obtain processed pixel information corresponding to each pixel, and each processed pixel generates a processed disturbed image.
[0134] Formula (1) is:
[0135]
[0136] Among them, output is the pixel information of each pixel on the perturbed image, ε is the set attack intensity, and δ is the pixel information of the processed pixel.
[0137] Exemplarily, if the size of the perturbed image is 112×112×3 (i.e., a 3-channel image), then for the first channel, the calculation process of ‖output‖2 can be: calculate the sum of the squares of the pixel information of each pixel on the first channel, and then take the square root of the sum of the squares to obtain the value of ‖output‖2 corresponding to the first channel; then divide the pixel information of each pixel on the first channel by the value of ‖output‖2, multiply the quotient by the set attack intensity ε, and obtain the pixel information of the processed pixel corresponding to each pixel, and then obtain the processed first channel. Based on the same processing process, the processed second channel and the processed third channel can be obtained, that is, the processed perturbed image is obtained.
[0138] When p is 1, if the size of the perturbed image is 112×112×3 (i.e., a 3-channel image), then for the first channel, the sum of the absolute values of the pixel information of each pixel on the first channel can be calculated, and then the pixel information of each pixel on the first channel is divided by the obtained sum of the absolute values, and the obtained quotient is multiplied by the set attack intensity ε to obtain the processed pixel information of each pixel corresponding to the processed pixel, and then the processed first channel is obtained. Based on the same processing process, the processed second channel and the processed third channel can be obtained, that is, the processed perturbed image is obtained.
[0139] When p is ∞, if the size of the perturbed image is 112×112×3 (i.e., a 3-channel image), the pixel range corresponding to the pixel point can be set. The pixel range corresponding to the pixel point can be set as needed. For example, the set pixel range can be [-8, 8]. Then, for the first channel, the pixel information of each pixel point on the first channel is determined. If the pixel information is within the set pixel range, the pixel information of the pixel point is not processed. For example, when the pixel information of pixel point A is 5, the pixel information of pixel point A is not processed. If the pixel information of the pixel point is greater than the maximum value of the pixel range, the pixel information of the pixel point is set to the maximum value. For example, if the pixel information of pixel point B is 10, the pixel information of pixel point B is set to 8. If the pixel information of the pixel point is less than the minimum value of the pixel range, the pixel information of the pixel point is set to the minimum value. For example, if the pixel information of pixel point C is -10, the pixel information of pixel point B is set to -8.
[0140] In a specific implementation, the attack intensity constraint module can be used to perform visible pixel weakening processing on pixel information of pixel points on the disturbed image to generate a processed disturbed image.
[0141] Then, based on the processed disturbed image and the face image to be processed, a processed face image can be generated. In specific implementation, the pixel information of the processed disturbed image and the corresponding pixel information on the face image to be processed can be summed to generate the processed face image.
[0142] Here, in order to reduce the influence of the visual features on the disturbed image on the face image to be processed, the pixel information of the pixels on the disturbed image can be firstly weakened to generate a processed disturbed image, and then based on the processed disturbed image and the face image to be processed, a processed face image is generated, so as to achieve the purpose of countering the attack while ensuring the display effect of the face image.
[0143] In an optional implementation, the processed face image containing disturbance information is generated based on a trained target neural network; the trained target neural network is obtained by the following steps:
[0144] Step 1: Obtain face image samples.
[0145] Step 2: Based on the face image sample and the neural network to be trained, determine the attack image sample corresponding to the face image sample.
[0146] Step three: based on the attack image sample, the face image sample, and the neural network to be trained, generate a processed face image sample.
[0147] Step 4: Based on the attack image samples and the processed face image samples, the neural network to be trained is trained, and / or based on the face image samples and the processed face image samples, the neural network to be trained is trained to obtain a trained target neural network.
[0148] Here, the target neural network may include an attack target selection neural network, a face image encoding module, a face image decoding module, and an attack intensity constraint module.
[0149] In a specific implementation, the face image sample can be any image sample containing face information. The face image sample is input into the target neural network, and the attack target selection neural network included in the target neural network determines the attack image sample corresponding to the face image sample, and the attack image sample is an image sample among the stored multiple candidate attack images; further, the face image encoding module, face image decoding module, and attack intensity constraint module included in the target neural network can process the face image sample and the attack image sample to generate a perturbation image sample corresponding to the face image sample; based on the perturbation image sample and the face image sample, a processed face image sample is generated.
[0150] Finally, a first loss value can be generated based on the attack image samples and the processed face image samples, a second loss value can be generated based on the face image samples and the processed face image samples, and the neural network to be trained is trained based on the first loss value and / or the second loss value to obtain a trained target neural network.
[0151] In an optional implementation, training the neural network to be trained based on the attack image sample and the processed face image sample includes:
[0152] 1. Extracting a first image feature corresponding to the processed face image sample and a second image feature corresponding to the attack image sample.
[0153] 2. Based on the first image feature and the second image feature, calculate a first loss value of the neural network to be trained.
[0154] 3. Use the first loss value to train the neural network to be trained.
[0155] Here, any trained face recognition neural network can be used to extract the first image feature corresponding to the processed face image sample and the second image feature corresponding to the attack image sample. Then the following formula (2) can be used to calculate the first loss value:
[0156] L1=1-cos(x adv ,x target ) (2)
[0157] Among them, x adv is the first image feature corresponding to the processed face image sample, x target is the second image feature corresponding to the attack image sample. Furthermore, the first loss value can be used to train the neural network to be trained, and after the trained target neural network is used to generate a processed face image, the first image feature corresponding to the processed face image sample is made similar to the second image feature corresponding to the attack image sample.
[0158] In the above implementation, a first loss value is generated and the neural network to be trained is trained using the first loss value to obtain a trained target neural network. A processed face image is generated based on the trained target neural network. This allows the facial feature information corresponding to the extracted processed face image to have the facial feature information of the target attack image, thereby achieving an effect of countering the attack.
[0159] In an optional implementation, training the neural network to be trained based on the face image sample and the processed face image sample includes:
[0160] 1. Extracting a first image feature corresponding to the processed face image sample and a third image feature corresponding to the face image sample.
[0161] 2. Based on the first image feature and the third image feature, calculate a second loss value of the neural network to be trained.
[0162] 3. Use the second loss value to train the neural network to be trained.
[0163] Here, any trained face recognition neural network can also be used to extract the first image feature corresponding to the processed face image sample and the third image feature corresponding to the face image sample. Then the first loss value can be calculated using the following formula (3):
[0164] L2=1+cos(x adv ,x in ) (3)
[0165] Among them, x adv is the first image feature corresponding to the processed face image sample, x in The third image feature corresponding to the face image sample. Furthermore, the second loss value can be used to train the neural network to be trained, and after the trained target neural network is used to generate a processed face image, the first image feature corresponding to the processed face image sample is dissimilar to the third image feature corresponding to the face image sample.
[0166] When the loss value includes a first loss value and a second loss value, the first loss value can be calculated using formula (2), and the second loss value can be calculated using formula (3). Based on the first loss value and the second loss value, the neural network to be trained is trained.
[0167] In the above implementation, by generating a second loss value, the neural network to be trained is trained using the second loss value to obtain a trained target neural network, and a processed facial image is generated based on the trained target neural network. This allows the facial feature information corresponding to the extracted processed facial image to be distinguished from the facial feature information of the facial image to be processed, thereby achieving an effect of counteracting attacks.
[0168] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0169] Based on the same concept, the present disclosure also provides an image protection device, see Figure 2 As shown, it is a schematic diagram of the architecture of the image protection device provided by the embodiment of the present disclosure, including an acquisition module 201, a selection module 202, and a generation module 203. Specifically:
[0170] An acquisition module 201 is used to acquire a face image to be processed;
[0171] A selection module 202 is used to obtain similarities between the face image to be processed and a plurality of candidate attack images, and select a target attack image from the candidate attack images based on the similarities;
[0172] The generation module 203 is used to process the face image to be processed based on the target attack image to generate a processed face image containing disturbance information; the disturbance information is used to interfere with the detection of real face information in the face image to be processed.
[0173] In a possible implementation, the selection module 202, when selecting a target attack image from the candidate attack images based on the similarity, is configured to:
[0174] From the multiple candidate attack images, the candidate attack image having the highest similarity to the face image to be processed is selected as the target attack image.
[0175] In a possible implementation, the selection module 202, when obtaining the similarity between the face image to be processed and a plurality of candidate attack images, is configured to:
[0176] Extracting facial features of the facial image to be processed;
[0177] Calculate the feature similarity between the facial features of the face image to be processed and the facial features corresponding to each of the multiple candidate attack images, and use the feature similarity as the similarity between the face image to be processed and the candidate attack images.
[0178] In a possible implementation manner, the stored multiple candidate attack images have multiple different facial attributes;
[0179] The facial attributes include, but are not limited to: skin color, gender, face orientation, face shape, and facial features.
[0180] In a possible implementation manner, the generating module 203, when processing the to-be-processed face image based on the target attack image to generate a processed face image containing disturbance information, is configured to:
[0181] Performing feature extraction on the target attack image to generate a first feature map corresponding to the target attack image, and performing feature extraction on the face image to be processed to generate a second feature map corresponding to the face image to be processed;
[0182] Based on the first feature map and the second feature map, a disturbance image is generated; the disturbance image contains disturbance information;
[0183] Based on the disturbance image and the face image to be processed, the processed face image is generated.
[0184] In a possible implementation, when generating the perturbed image based on the first feature map and the second feature map, the generating module 203 is configured to:
[0185] Extracting difference features from a combined feature map of the first feature map and the second feature map to obtain a difference feature map representing difference information between the first feature map and the second feature map;
[0186] Based on the difference feature map, the perturbed image is generated.
[0187] In a possible implementation manner, the generating module 203, when generating the processed facial image based on the disturbed image and the facial image to be processed, is configured to:
[0188] Performing visible pixel weakening processing on pixel information of pixel points on the disturbed image to generate a processed disturbed image;
[0189] Based on the processed disturbance image and the face image to be processed, the processed face image is generated.
[0190] In a possible implementation manner, the processed face image containing disturbance information is generated based on a trained target neural network; the apparatus further comprises: a training module 204, configured to obtain the trained target neural network through the following steps:
[0191] Obtain face image samples;
[0192] Based on the face image sample and the neural network to be trained, determining an attack image sample corresponding to the face image sample;
[0193] Based on the attack image sample, the face image sample, and the neural network to be trained, generating a processed face image sample;
[0194] Based on the attack image samples and the processed face image samples, the neural network to be trained is trained, and / or based on the face image samples and the processed face image samples, the neural network to be trained is trained to obtain a trained target neural network.
[0195] In a possible implementation, the training module 204, when training the neural network to be trained based on the attack image sample and the processed face image sample, is configured to:
[0196] Extracting a first image feature corresponding to the processed face image sample and a second image feature corresponding to the attack image sample;
[0197] Calculating a first loss value of the neural network to be trained based on the first image feature and the second image feature;
[0198] The neural network to be trained is trained using the first loss value.
[0199] In a possible implementation, the training module 204, when training the neural network to be trained based on the face image sample and the processed face image sample, is configured to:
[0200] Extracting a first image feature corresponding to the processed face image sample and a third image feature corresponding to the face image sample;
[0201] Calculating a second loss value of the neural network to be trained based on the first image feature and the third image feature;
[0202] The neural network to be trained is trained using the second loss value.
[0203] In some embodiments, the functions or templates contained in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0204] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 3 As shown, it is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, including a processor 301, a memory 302, and a bus 303. Among them, the memory 302 is used to store execution instructions, including a memory 3021 and an external memory 3022; the memory 3021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 301, and the data exchanged with the external memory 3022 such as a hard disk. The processor 301 exchanges data with the external memory 3022 through the memory 3021. When the electronic device 300 is running, the processor 301 communicates with the memory 302 through the bus 303, so that the processor 301 executes the following instructions:
[0205] Obtain the face image to be processed;
[0206] Selecting a target attack image from the stored plurality of candidate attack images based on a similarity between each candidate attack image and the face image to be processed;
[0207] Based on the target attack image and the face image to be processed, a processed face image containing disturbance information is generated; the disturbance information is used to interfere with the detection of real face information.
[0208] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image protection method described in the above method embodiment are executed.
[0209] The computer program product of the image protection method provided in the embodiment of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the image protection method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.
[0210] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0211] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0212] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0213] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0214] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. An image protection method, It is characterized in that include: Obtain the face image to be processed; Acquire the similarity between the face image to be processed and a plurality of candidate attack images, and select a target attack image from the candidate attack images based on the similarity; Processing the face image to be processed based on the target attack image to generate a processed face image containing disturbance information; the disturbance information is used to interfere with the detection of real face information in the face image to be processed; The processing of the face image to be processed based on the target attack image to generate a processed face image containing disturbance information includes: extracting features from the target attack image to generate a first feature map corresponding to the target attack image, and extracting features from the face image to be processed to generate a second feature map corresponding to the face image to be processed; generating a disturbance image based on the first feature map and the second feature map; the disturbance image contains disturbance information; Based on the disturbance image and the face image to be processed, the processed face image is generated.
2. The method according to claim 1, It is characterized in that The selecting a target attack image from the candidate attack images based on the similarity comprises: From the multiple candidate attack images, the candidate attack image having the highest similarity to the face image to be processed is selected as the target attack image.
3. The method according to claim 1 or 2, It is characterized in that The obtaining of the similarity between the face image to be processed and a plurality of candidate attack images includes: Extracting facial features of the facial image to be processed; Calculate the feature similarity between the facial features of the face image to be processed and the facial features corresponding to each of the multiple candidate attack images, and use the feature similarity as the similarity between the face image to be processed and the candidate attack images.
4. The method according to any one of claims 1 to 3, It is characterized in that The plurality of candidate attack images have a plurality of different facial attributes; The facial attributes include, but are not limited to: skin color, gender, face orientation, face shape, and facial features.
5. The method according to claim 1, It is characterized in that Generating a perturbed image based on the first feature map and the second feature map includes: Extracting difference features from a combined feature map of the first feature map and the second feature map to obtain a difference feature map representing difference information between the first feature map and the second feature map; Based on the difference feature map, the perturbed image is generated.
6. The method according to claim 1, It is characterized in that Generating the processed face image based on the disturbance image and the face image to be processed includes: Performing visible pixel weakening processing on pixel information of pixel points on the disturbed image to generate a processed disturbed image; Based on the processed disturbance image and the face image to be processed, the processed face image is generated.
7. The method according to any one of claims 1 to 6, It is characterized in that The processed face image containing disturbance information is generated based on a trained target neural network; the trained target neural network is obtained by the following steps: Obtain face image samples; Based on the face image sample and the neural network to be trained, determining an attack image sample corresponding to the face image sample; Based on the attack image sample, the face image sample, and the neural network to be trained, generating a processed face image sample; Based on the attack image samples and the processed face image samples, the neural network to be trained is trained, and / or based on the face image samples and the processed face image samples, the neural network to be trained is trained to obtain a trained target neural network.
8. The method according to claim 7, It is characterized in that Based on the attack image sample and the processed face image sample, training the neural network to be trained includes: Extracting a first image feature corresponding to the processed face image sample and a second image feature corresponding to the attack image sample; Calculating a first loss value of the neural network to be trained based on the first image feature and the second image feature; The neural network to be trained is trained using the first loss value.
9. The method according to claim 7, It is characterized in that Training the neural network to be trained based on the face image sample and the processed face image sample includes: Extracting a first image feature corresponding to the processed face image sample and a third image feature corresponding to the face image sample; Calculating a second loss value of the neural network to be trained based on the first image feature and the third image feature; The neural network to be trained is trained using the second loss value.
10. An image protection device, It is characterized in that include: An acquisition module, used for acquiring a face image to be processed; A selection module, configured to obtain similarities between the face image to be processed and a plurality of candidate attack images, and select a target attack image from the candidate attack images based on the similarities; A generating module, configured to process the face image to be processed based on the target attack image to generate a processed face image containing disturbance information; the disturbance information is used to interfere with the detection of real face information in the face image to be processed; The generating module, when processing the face image to be processed based on the target attack image to generate a processed face image containing disturbance information, is used to: extract features from the target attack image to generate a first feature map corresponding to the target attack image, and extract features from the face image to be processed to generate a second feature map corresponding to the face image to be processed; Based on the first feature map and the second feature map, a disturbed image is generated; the disturbed image contains disturbance information; based on the disturbed image and the face image to be processed, the processed face image is generated.
11. An electronic device, It is characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image protection method as described in any one of claims 1 to 9 are performed.
12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image protection method according to any one of claims 1 to 9 are executed.
Citation Information
Patent Citations
Methods and systems for content processing
CA2734613A1
Encrypted face recognition method based on ciphertext domain
CN107045627A
