Face image processing method, related apparatus, and storage medium

Abstract

Embodiments of the present application relate to the field of data security, and provide a face image processing method, related apparatus and storage medium, the method comprising: extracting face original features and intermediate layer original features from a first face image; inputting a candidate adversarial sample obtained by adding a target disturbance to the first face image into a target model to extract face adversarial features and intermediate layer adversarial features; calculating a first similarity between average face features and the face adversarial features, a second similarity between the face original features and the face adversarial features, and a third similarity between the intermediate layer original features and the intermediate layer adversarial features; obtaining a target gradient by performing reverse derivation on the candidate adversarial sample; optimizing the three similarities, updating the disturbance in the candidate adversarial sample according to the target gradient, and inputting the candidate adversarial sample into the target model as a candidate adversarial sample until a similarity between a target adversarial sample and the first face image is lower than a first threshold. The present scheme can improve the misidentification effect of the adversarial sample and protect user privacy data.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data security, and in particular to a face image processing method, related device and storage medium. BACKGROUND

[0002] With the popularity of the Internet, while it is convenient for people to obtain more information, personal privacy information is also more likely to be leaked. For example, in the information system of a modern enterprise, in the case of needing to publicly disclose personal face images, the publicly disclosed face images are vulnerable to attacks and cause leakage of personal privacy information. In order to resist attacks, encryption is generally performed on the face images to form, for example, face images with watermarks.

[0003] In the research and practice of the prior art, the inventors of the present application found that the encryption method used in the prior art needs to determine some features of a first face image first, and then encrypts the face image according to the features, resulting in a slow speed of generating encrypted images. SUMMARY

[0004] Embodiments of the present application provide a face image processing method, related device and storage medium, which can improve the misidentification effect of adversarial samples and protect user privacy data.

[0005] In a first aspect, embodiments of the present application provide a face image processing method from the perspective of a business server, which includes:

[0006] extracting face original features from a first face image to be processed and intermediate layer original features of a target model;

[0007] adding target perturbations to the first face image to obtain a candidate adversarial sample;

[0008] inputting the candidate adversarial sample into the target model to extract face adversarial features and intermediate layer adversarial features;

[0009] calculating a first similarity between a preset average face feature and the face adversarial features, a second similarity between the face original features and the face adversarial features, and a third similarity between the intermediate layer original features and the intermediate layer adversarial features;

[0010] if the first similarity is less than a first preset threshold, the second similarity is greater than a second preset threshold, and the third similarity is greater than a third preset threshold, then performing reverse derivation on the candidate adversarial sample to obtain a target gradient;

[0011] The first similarity is minimized, the second similarity and the third similarity are maximized, the perturbation in the candidate adversarial sample is updated according to the target gradient, and the updated target adversarial sample is input into the target model as the candidate adversarial sample until the similarity between the obtained target adversarial sample and the first face image is lower than the first threshold.

[0012] In some embodiments, the method further includes:

[0013] If the first similarity is greater than the first preset threshold, the second similarity is less than the second preset threshold, and the third similarity is less than the second preset threshold, then the candidate adversarial sample is taken as the target adversarial sample.

[0014] In some implementations, updating the perturbation in the candidate adversarial example according to the target gradient includes:

[0015] Update the pixel value of each pixel in the candidate adversarial sample according to the target gradient.

[0016] In some implementations, the first face image is derived from at least one of the following methods:

[0017] The target user has specific event information to be posted on a social media platform, and the specific event information includes at least one initial facial image;

[0018] Alternatively, a set of historical images posted by the target user on a social media platform within a historical period, the set of historical images including at least one unprocessed candidate historical image of the target user, the candidate historical images including images of the target user that meet preset privacy protection conditions.

[0019] In some implementations, the first face image is derived from at least one of the following methods:

[0020] The target user's historical image set to be uploaded to financial or government platforms;

[0021] Alternatively, the target user's historical image set uploaded to financial or government platforms during a historical period;

[0022] The historical image set includes at least one unprocessed candidate historical image of the target user.

[0023] In some implementations, acquiring the first face image to be processed includes:

[0024] The business server determines the number of inputs to the target model, and according to the number of inputs, inputs each of the first face images as an input to the target model to obtain the target adversarial samples corresponding to each first face image.

[0025] In some implementations, acquiring the first face image to be processed includes:

[0026] Determine the number of inputs to the target model;

[0027] According to the number of inputs, each sub-image in the first face image is input to the target model as an input to obtain the target adversarial sample, wherein each input corresponds to a pixel region in the first face image.

[0028] Secondly, embodiments of this application provide a face image processing method from the perspective of a search engine, the method comprising:

[0029] Receive a search instruction, which includes key information about the target user;

[0030] In response to the search command, a set of candidate images matching the key information is obtained from the business server; wherein, the set of candidate images includes at least one first face image and at least one target adversarial sample, wherein the target adversarial sample is obtained by inputting the first face image into a target model; the similarity between the target adversarial sample and the first face image is lower than a first threshold;

[0031] The target image set is determined from the candidate image set as the search results for the target user, and the target image set does not include target adversarial samples.

[0032] In some implementations, the target image set refers to images that have been posted by at least one target user and include at least one human face.

[0033] In some implementations, determining the target image set as the search results for the target user from the candidate image set includes:

[0034] The candidate image set is input into the face recognition model to obtain the first similarity between the target adversarial sample and the second face image, and the second similarity between each first face image and the second face image; the first similarity is lower than the first threshold, the second similarity is not less than the third threshold, and the first threshold is less than the third threshold;

[0035] The first face image selected from the candidate image set, whose similarity to the second face image is not less than the third threshold, is selected as the target image set.

[0036] Thirdly, embodiments of this application provide an image processing apparatus having the function of implementing the face image processing method corresponding to the first aspect described above. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, and the modules can be software and / or hardware.

[0037] In some embodiments, the image processing apparatus includes:

[0038] The input / output module is used to acquire the first face image to be processed.

[0039] The processing module is used to extract the original facial features and the intermediate layer original features of the target model from the first facial image.

[0040] Add a target perturbation to the first face image to obtain candidate adversarial examples;

[0041] The candidate adversarial samples are input into the target model to extract face adversarial features and intermediate layer adversarial features;

[0042] Calculate the first similarity between the preset average face feature and the adversarial face feature, calculate the second similarity between the original face feature and the adversarial face feature, and calculate the third similarity between the original intermediate layer feature and the adversarial intermediate layer feature;

[0043] If the first similarity is less than the first preset threshold, the second similarity is greater than the second preset threshold, and the third similarity is greater than the third preset threshold, then the candidate adversarial sample is back-derived to obtain the target gradient.

[0044] The first similarity is minimized, the second similarity and the third similarity are maximized, the perturbation in the candidate adversarial sample is updated according to the target gradient, and the updated target adversarial sample is input into the target model as the candidate adversarial sample until the similarity between the obtained target adversarial sample and the first face image is lower than the first threshold.

[0045] Fourthly, embodiments of this application also provide a search engine for implementing the face image acquisition method described in the second aspect above, the search engine comprising:

[0046] The transceiver module is used to receive search instructions, which include key information about the target user.

[0047] The processing module is configured to, in response to the search command, obtain a set of candidate images matching the key information from the business server through the transceiver module; wherein, the set of candidate images includes at least one first face image and at least one target adversarial sample, wherein the target adversarial sample is obtained by inputting the first face image into a target model; the similarity between the target adversarial sample and the first face image is lower than a first threshold.

[0048] The processing module is further configured to determine a target image set from the candidate image set as the search results for the target user, wherein the target image set does not include target adversarial samples.

[0049] In some implementations, the first face image refers to an image that has been posted by at least one target user and includes at least one face.

[0050] In some implementations, the processing module is specifically used for:

[0051] The candidate image set is input into the face recognition model to obtain the first similarity between the target adversarial sample and the second face image, and the second similarity between each first face image and the second face image; the first similarity is lower than the first threshold, the second similarity is not less than the third threshold, and the first threshold is less than the third threshold;

[0052] The first face image selected from the candidate image set, whose similarity to the second face image is not less than the third threshold, is selected as the target image set.

[0053] Fifthly, embodiments of this application provide a face image processing apparatus, the face image processing apparatus comprising: at least one processor and a memory; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the steps in the first aspect, any embodiment of the first aspect, or any face image processing method provided in the second aspect.

[0054] Sixthly, embodiments of this application provide a computer-readable storage medium having the function of implementing the face image processing method corresponding to the first aspect described above. The function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, and the modules can be software and / or hardware. Specifically, the computer-readable storage medium stores multiple instructions adapted for loading by a processor to execute the steps in the first aspect, any implementation of the first aspect, or any face image processing method provided in the second aspect of this application.

[0055] Compared to existing technologies, the solution provided in this application involves performing N iterations on candidate adversarial samples obtained by adding perturbations to the first face image based on the target model. The perturbations in the candidate adversarial samples are continuously adjusted in each iteration until the loss function converges. This indicates that the target adversarial sample obtained after processing the first face image by the target model is different from the first face image before processing by the target model, thus allowing the final target adversarial sample to be used as a substitute for the first face image. Therefore, on one hand, by minimizing the first similarity, the similarity between the real face and the average face can be improved. This makes the recognition result of similar face recognition models (such as models deployed in search engines) more likely to resemble a "generalized average face." The higher the similarity to the average face, the less likely it is to identify the true identity of the target user, i.e., it is impossible to conclude that the real user corresponding to the target adversarial sample is the target user. On the other hand, maximizing the second and third similarities can reduce the actual similarity of the output, so that similar face recognition models (such as those deployed in search engines) tend to have lower similarity results for the adversarial sample, meaning that it is impossible to conclude that the real user corresponding to the adversarial sample is the target user.

[0056] Therefore, on the one hand, once the adversarial sample is published on a business server (such as a social platform), even if a search engine crawls the adversarial sample, it will misidentify it due to its adversarial nature, or be unable to bind it to the identity information of the target user being searched. This effectively protects the target user's privacy data from malicious collection or exploitation. On the other hand, once the first face image uploaded to the business server (such as social platforms, financial platforms, government platforms, etc., which require uploading privacy images including the target user's face) is updated with the adversarial sample, even if the adversarial sample uploaded to the business server is leaked for some reason, or is crawled by a search engine after being leaked, it will misidentify the adversarial sample due to its adversarial nature, or be unable to bind it to the identity information of the target user being searched. This effectively protects the target user's privacy data from malicious collection or exploitation.

[0057] Furthermore, because the third similarity is maximized, this adversarial example does not cause visual interference for the user; to the user, it appears as the same image. However, from a machine vision perspective, this adversarial example cannot be directly identified by an intelligent search engine based on an image recognition model and cannot be linked to the target user's identity information. Therefore, the embodiments of this application can effectively protect the privacy data of the target user. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of an application environment provided in the embodiments of this application;

[0059] Figure 2 This is a flowchart illustrating a face image processing method in an embodiment of this application.

[0060] Figure 3 This is a schematic diagram of a model structure for generating target adversarial examples in an embodiment of this application;

[0061] Figure 4a This is a schematic diagram of adding adversarial perturbation to a first face image to obtain a target adversarial sample in an embodiment of this application;

[0062] Figure 4b This is a schematic diagram illustrating the addition of perturbation to the facial image of the target user to be posted on Weibo in an embodiment of this application;

[0063] Figure 4c This is a schematic diagram illustrating the addition of perturbation to a complete face image synthesized from a nine-grid image uploaded to a social media platform, as described in this application embodiment.

[0064] Figure 4d This is a schematic diagram illustrating the addition of perturbation to the facial images of target users historically posted on social media platforms, as described in this application embodiment.

[0065] Figure 4e This is a schematic diagram illustrating the addition of perturbation to facial images of target users and specific users historically posted on social media platforms, as described in this application embodiment.

[0066] Figure 5 This is a flowchart illustrating an image acquisition method in an embodiment of this application.

[0067] Figure 6 This is a schematic diagram of a scenario for the image acquisition method in an embodiment of this application;

[0068] Figure 7 This is a schematic diagram of the structure of an image processing device in an embodiment of this application;

[0069] Figure 8 This is a schematic diagram of the structure of a search device in an embodiment of this application;

[0070] Figure 9 This is a schematic diagram of the structure of a physical device implementing the face image processing method in the embodiments of this application;

[0071] Figure 10 This is a schematic diagram of the structure of a mobile phone implementing the face image processing method in an embodiment of this application;

[0072] Figure 11This is a schematic diagram of the structure of a server implementing the face image processing method in an embodiment of this application. Detailed Implementation

[0073] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects (e.g., the first threshold and the second threshold in the embodiments of this application represent thresholds of different values), and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules in the embodiments of this application is merely a logical division; in actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not performed. Additionally, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, and the indirect coupling or communication connection between modules may be electrical or other similar forms, none of which are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0074] This application provides a face image processing method, related apparatus, and storage medium, which can be used on servers or terminal devices. Specifically, it can be used to perform privacy protection processing on images to be published and previously published, thereby generating face adversarial examples. This makes it impossible for a search engine (i.e., a search device) based on a neural network model to directly identify the first face image matching the specific user even after crawling face images of a specific user from the network (e.g., crawling face images previously published by the target user from a business server) and preprocessing the face images.

[0075] In some implementations, this solution is applied to, for example... Figure 1 The communication system shown may include at least one service server and at least one search device. Figure 1Only one search device (a) is shown in the diagram. This search device can be a server or a terminal. When it is a terminal, it has a client installed. The clients installed on each search device can be the same or different versions of the client, or clients with the same or different operating systems. This application does not limit the version of the client or the type of operating system corresponding to the client. A detailed description follows.

[0076] Business server: This server can be used to provide image search services to various search devices, as well as to provide users with image upload, download, or preprocessing services. For example, the server can perform anti-disturbance processing on images uploaded by users and then update them online. Image processing devices can be deployed on this business server.

[0077] Search device: It can crawl facial images or videos that match the key information of the target user from the business server, and then preprocess the images or videos before recognizing the preprocessed images to search for images that include the face of the target user.

[0078] When based on such Figure 1 When the communication system described above implements the face image processing method and face image acquisition method, the service server can perform privacy protection processing on the face image 'a' uploaded by the target user (for example, inputting face image 'a' into a target model based on unsupervised learning to add adversarial perturbations to face image 'a' to obtain target adversarial examples), and then publish the target adversarial examples online after updating them. On the one hand, this makes it impossible for the search device to identify that face image 'a' belongs to the target user after crawling face image 'a' from the service server, thus interfering with the face recognition function of the search device. On the other hand, since the service server adds small perturbations to the cheeks and forehead and larger perturbations to other locations during the face adversarial example generation process, it can effectively reduce the impact of adversarial perturbations on the visual effect of the face.

[0079] The solutions provided in this application involve technologies such as Artificial Intelligence (AI), Natural Language Processing (NLP), and Machine Learning (ML), which are specifically illustrated through the following embodiments:

[0080] AI, or Artificial Intelligence, refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, Artificial Intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence. Artificial Intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0081] AI technology is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0082] NLP is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. Natural Language Processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close connection with linguistic research. Natural Language Processing techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0083] In the field of artificial intelligence, specifically for facial recognition, this application can generate a general perturbation with high attack robustness using a set of facial images. By adding this general perturbation to the facial images in the set, the image classification model is unable to accurately recognize the perturbated facial images, thereby achieving an attack on the image classification model.

[0084] It should be specifically noted that the servers involved in this application's embodiments (e.g., business servers, search engines) can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The image processing devices involved in this application's embodiments can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, personal digital assistants, etc., but are not limited to these. The image processing devices and the server can be directly or indirectly connected via wired or wireless communication, and this application's embodiments do not impose any limitations on this connection.

[0085] When a user's portrait is disseminated on public social networks, the user does not want third parties to be able to use web crawlers, search engines, or other means to search for their own facial image using AI image recognition technology, such as facial recognition. Therefore, they need to add adversarial perturbations to the image to interfere with facial recognition. At the same time, the user wants their portrait to be viewed in their social circle without affecting viewers' judgment of their appearance; therefore, the user does not want the adversarial perturbations to excessively affect the visual effect of their appearance. To address this, the embodiments of this application mainly adopt the following technical solutions:

[0086] Images to be published and previously published images are input into a pre-trained target model to perform privacy protection processing on the images, thereby generating corresponding adversarial examples. This prevents a search engine based on a neural network model from directly identifying the first face image matching a specific user even after crawling facial images of the target user from the web (e.g., crawling previously published facial images of the target user from a business server) and preprocessing the facial images. Specifically, the target model adds perturbations to the candidate adversarial examples obtained by adding perturbations to the facial images to be processed, and iterates N times. By continuously adjusting the perturbations in the candidate adversarial examples in each iteration, the loss function converges. This means that the target adversarial example obtained after processing the first face image by the target model is different from the first face image before processing by the target model, and thus the final target adversarial example can be used as a substitute for the first face image.

[0087] The following combination Figures 2-10 The technical solutions of the embodiments of this application will be described by way of example.

[0088] Since the face image processing method in this embodiment is based on a pre-trained target model to process the first face image, the target model is introduced before describing the face image processing method. This target model is a model retrained on a private face dataset using the open-source face recognition model Arcface. The face recognition algorithm in this embodiment incorporates at least one target model and at least one average face image. (Average face image: Face) avg It is obtained by averaging N (e.g., 800,000) private face datasets. One calculation formula is as follows:

[0089]

[0090] To enable the target model to perform privacy protection processing on the first face image, the target model in this embodiment is retrained based on the aforementioned average face image. Specifically, when the average face image is input into the face recognition model, the face recognition model first extracts the original facial features from the image.ori and the original features of the intermediate layer of the model inter_ori And simultaneously extract average face features avg Save the three features for later use. Figure 2 This is used when calculating the loss in the corresponding embodiments (e.g., calculating the first similarity based on the first loss function, calculating the second similarity based on the second loss function, and calculating the third similarity based on the third loss function).

[0091] This application does not limit the selection or structure of the target model or simulation model.

[0092] As described in the above embodiments, after obtaining the target model by retraining the aforementioned face recognition model, the face image to be processed (e.g., the first face image) can be processed based on the target model to obtain adversarial examples (e.g., target adversarial examples) that meet privacy protection conditions. Specifically, see... Figure 2 A face image processing method is provided, which is implemented based on an unsupervised learning target model. Taking adding a watermark to a face image for anti-search purposes as an example, this method can be executed by a business server. This business server can be a social platform, financial platform, government platform, retrieval system, or other platform that needs to upload privacy images including the face of a target user. This application embodiment does not limit this. The embodiments of this application include:

[0093] 201. Obtain the first face image to be processed.

[0094] The first facial image is a facial image that requires privacy protection, including but not limited to the facial image of the target user and facial images of specific users. The specific user includes facial images of users associated with the target user (e.g., friends), facial images of any user specified by the target user, and facial images that meet preset privacy protection conditions as defined by the business server. This application embodiment does not limit this specificity. This application embodiment also does not limit the number of first facial images in the input target model or the user corresponding to each first facial image.

[0095] In some implementations, since the business server can provide various business services to users, and the business server can be a social platform, financial platform, government platform, or search platform, the source of the first facial image that the business server can process also varies. Specifically, it can be explained by data leakage or crawling paths:

[0096] (1) Targeting the crawling of first facial images posted by users on social media platforms

[0097] In this case, the first facial image may originate from at least one of the following methods:

[0098] a1. Specific event information that the target user is about to publish on a social media platform.

[0099] The specific event information includes at least one initial face image.

[0100] For example, user A is editing a Weibo post (a) on Sina Weibo but has not yet published it. This Weibo post (a) includes multiple facial images uploaded by user A. These multiple facial images uploaded by user A may include only user A's facial image, only user B's facial image, or both user A's and user B's facial images simultaneously. This application's embodiments do not limit this. A schematic diagram can be referred to as follows: Figure 4b I will not elaborate further.

[0101] b1. The set of historical images that the target user has published on social media platforms within a historical period.

[0102] The historical image set includes at least one unprocessed candidate historical image of the target user, and the candidate historical image includes images of the target user that meet preset privacy protection conditions.

[0103] For example, user A had posted 20 Weibo posts on Sina Weibo before February 27, 2022 (referred to as historical Weibo posts). Two of these historical Weibo posts (e.g., Weibo post b and Weibo post c) include multiple facial images uploaded by user A (referred to as historical facial images). Similarly, the historical facial images uploaded by user A may include only user A's facial image, only user B's facial image, or both user A's and user B's facial images. This application embodiment does not limit this. A schematic diagram can be referred to as follows: Figure 4d I will not elaborate further.

[0104] As can be seen, the embodiments of this application can perform privacy protection processing on images to be published and published from various channels, so that even if a face image related to one's privacy is crawled to the local search engine by a search engine based on a face recognition model after being published on a social platform, it is likely to be filtered by the search engine, that is, it can resist the search of a search engine based on a face recognition model to a certain extent.

[0105] (2) Data leaked from financial or government platforms, and the first facial image of the leaked data crawled.

[0106] a2. The set of target images to be uploaded by the target user on the financial or government platform.

[0107] The target image set can be a separately uploaded image set or an image set included in authentication or other related materials. This application embodiment does not limit this, and other embodiments are similar and will not be described in detail.

[0108] For example, when user B performs identity binding or verification services on financial platform A, user B needs to upload at least one set of multiple facial images of user B on financial platform A. These multiple facial images of user B can be ID photos, residence permit photos, passport photos, registration photos, or other document-type photos.

[0109] b2. Historical image sets uploaded by the target user to financial or government platforms within a historical period.

[0110] The historical image set includes at least one unprocessed candidate historical image of the target user.

[0111] As can be seen, the embodiments of this application can perform privacy protection processing on images to be published and published from various channels, so that even if facial images related to one's privacy are uploaded to financial platforms or government platforms and leaked to the Internet for some reason, they are likely to be filtered by search engines, that is, they can resist search by search engines based on facial recognition models to a certain extent.

[0112] In other embodiments, the first face image can be a sub-image from the entire initial face image, or it can be the acquired initial face image. That is, the entire initial face image can be input into the target model to generate adversarial examples, or a sub-image from the entire initial face image can be input into the target model to generate adversarial examples. For example, when the initial face image includes only at least one face image of the target user, a sub-image of the target user's face image can be obtained from it; or when the initial face image includes the target user's face image and face images of other users, the initial face image can be divided to obtain the target user's face image. The specific method for obtaining the first face image to be processed is as follows:

[0113] Obtain the initial face image;

[0114] The initial face image is divided into regions to obtain multiple sub-images;

[0115] One or all of the sub-images among the plurality of sub-images are used as the first face image.

[0116] The plurality of sub-images include at least one first region and at least one second region, wherein the first region is a non-facial key feature region, such as the forehead, chin, and cheeks. The second region is a facial key feature region, such as facial features such as eyebrows, eyes, nose, mouth, and ears.

[0117] As can be seen, the embodiments of this application fully consider the users corresponding to the sub-images included in the initial face image. By preprocessing the initial face image, it is possible to effectively improve the subsequent rapid and targeted generation of adversarial samples based on the preprocessed first face image.

[0118] This application does not limit the shape, granularity, method, number, or area of ​​the first and second regions. The face region contains multiple non-facial key feature regions and facial key feature regions. In this application embodiment, the region can be divided according to actual needs (e.g., business requirements, accuracy, etc.). In some implementations, only at least one first region may be divided, i.e., a region specifically used to add the first perturbation; this is not limited.

[0119] 202. Extract the original facial features and the intermediate layer original features of the target model from the first facial image.

[0120] Specifically, the first face image can be input into the target model to extract the original face features and the original features of the intermediate layer of the target model.

[0121] Among them, the original facial features are the original facial features of the target user in the first facial image, without any processing.

[0122] The intermediate layer original feature of the target model refers to an output after the first face image enters the target model and is processed in the convolutional layer of the target model.

[0123] Average face features refer to the facial features of an average face, for example, the facial features of an average face in Hunan Province.

[0124] An average face is created by extracting facial features from a number of ordinary faces, averaging the measured data, and then using computer technology to generate a composite face that is considered attractive. Because different cultures have different standards of beauty, aesthetic preferences and their respective "average faces" vary from place to place. The attractiveness of an average face depends on the overall quality of the data provided. Generally, an average face is more attractive than the average overall level. An average face is considered beautiful, but not necessarily the most beautiful, because the average face of a beautiful group is more attractive than the average face of an average person. Then, the average contour obtained in the first step is filled with the average skin tone of the corresponding locations. With a sufficient number of samples, a relatively accurate "average face" can be generated for that group.

[0125] A target adversarial example refers to a face image obtained after processing a first face image through a target model. This target adversarial example ensures that the user's visual perception does not affect the result of identifying it as the first face image, nor does it affect the visual effect of the first face image. Furthermore, it prevents AI-based search engines from identifying the target user and binding it to the target user's identity even if they crawl the target adversarial example. In other words, it is useless for the search engine to crawl the target adversarial example.

[0126] 203. Add target perturbation to the first face image to obtain candidate adversarial samples.

[0127] For example, the algorithm adds some random perturbations to the original image to create an adversarial example adv, and then starts a loop N times.

[0128] 204. Input the candidate adversarial samples into the target model to extract face adversarial features and intermediate layer adversarial features.

[0129] In this embodiment of the application, in each loop, the target model first extracts face adversarial features from the adversarial examples. adv Adversarial features in intermediate layers inter_adv .

[0130] 205. Calculate the first similarity between the preset average face feature and the adversarial face feature, calculate the second similarity between the original face feature and the adversarial face feature, and calculate the third similarity between the original intermediate layer feature and the adversarial intermediate layer feature.

[0131] In some implementations, a first similarity between the average face features and the adversarial face features can be calculated based on a first loss function, a second similarity between the original face features and the adversarial face features can be calculated based on a second loss function, and a third similarity between the original intermediate layer features and the adversarial face features can be calculated based on a third loss function.

[0132] In some implementations, the first loss function calculates the feature. avg and feature adv The Euclidean distance between them. Its formula is:

[0133]

[0134] Then, the second loss function is used to calculate the original facial features. ori Adversarial facial features adv The formula for the cosine distance between them is:

[0135]

[0136] Finally, the third loss function is used to calculate the original features of the intermediate layer. inter_ori Adversarial features of the intermediate layer

[0137] feature inter_adv The formula for the cosine distance between them is:

[0138]

[0139] 206. If the first similarity is less than the first preset threshold, the second similarity is greater than the second preset threshold, and the third similarity is greater than the third preset threshold, then the candidate adversarial sample is back-derived to obtain the target gradient.

[0140] The values ​​of the first preset threshold, the second preset threshold, and the third preset threshold are not limited.

[0141] After calculating the above three loss values, the similarity of each value can be compared with a preset threshold. If the first similarity is greater than the first preset threshold, it indicates that although a candidate adversarial example has been obtained after adding perturbation to the first face image, the first similarity between the adversarial features of the candidate adversarial example and the average face features is still low, indicating insufficient similarity. To prevent the search engine from recognizing it from machine vision based on the face recognition model, the similarity between the adversarial features of the candidate adversarial example and the average face features needs to be increased. Similarly, if the second similarity between the original face features and the adversarial face features is too high, and the third similarity between the original features of the intermediate layer and the adversarial features of the intermediate layer is also too high, it is easy for the search engine to fail to recognize it from machine vision based on the face recognition model. Therefore, another round of iteration is required, that is, it is necessary to continue iterating to obtain a better perturbated target adversarial example. In this embodiment, the gradient of each pixel value of the adversarial example can be calculated using the backpropagation algorithm.

[0142] 207. Minimize the first similarity, maximize the second similarity and the third similarity, update the perturbation in the candidate adversarial sample according to the target gradient, and input the updated target adversarial sample as the candidate adversarial sample into the target model until the similarity between the obtained target adversarial sample and the first face image is lower than the first threshold.

[0143] The adversarial sample refers to the face image obtained after the first face image is processed by the target model. This adversarial sample is designed to ensure that it does not affect the user's visual perception of the first face image, nor does it affect the visual effect of the first face image. Furthermore, it is designed to prevent AI-based search engines from identifying the adversarial sample as the target user from a machine vision perspective and from binding it to the target user's identity. In other words, it is useless for the search engine to crawl the adversarial sample.

[0144] Specifically, the first loss function, the second loss function, and the third loss function can be optimized separately. For example, the first loss function can be minimized, and the second and third loss functions can be maximized. The total loss can be obtained using the following formula:

[0145] loss = -loss1 + loss2 + loss3

[0146] Then, the pixel values ​​of each pixel in the candidate adversarial sample are updated according to the target gradient, and the updated target adversarial sample is used as the first face image and input into the target model until the similarity between the obtained target adversarial sample and the first face image is lower than a first threshold.

[0147] In this embodiment, on the one hand, minimizing the first loss function can improve the similarity between the real face and the average face, making the recognition result of similar face recognition models (such as those deployed in search engines) more likely to be a "generalized average face". The higher the similarity to the average face, the less likely it is to identify the real identity of the target user, i.e., it is impossible to conclude that the real user corresponding to the target adversarial sample is the target user. On the other hand, maximizing the second loss function can reduce the actual similarity, making the recognition result of similar face recognition models (such as those deployed in search engines) more likely to have a lower similarity, i.e., it is impossible to conclude that the real user corresponding to the target adversarial sample is the target user.

[0148] In some implementations, only the first loss function may be minimized, and the second loss function may be maximized. Maximizing the third loss function aims to make the final output adversarial example more visually appealing. Because the third similarity is maximized, the adversarial example does not cause recognition interference to the user; to the user, it is the same image. However, from the perspective of machine vision, this adversarial example cannot be directly recognized by an intelligent search engine based on image recognition models and cannot be linked to the target user's identity information.

[0149] In some implementations, the Adam[5] optimization algorithm is used to minimize loss1, and the Ti-Mi-FGSM[6] algorithm is used to maximize loss2 and loss3. Then the value of each pixel in the target adversarial sample is updated, and the next loop is entered after completion.

[0150] After N iterations, the target model outputs a face image 〖Face〗_adv with adversarial perturbation added. This image is created by adding a perturbation with a maximum amplitude not exceeding ε to the original image. Our minimum value for ε is 9, which has almost no impact on the visual effect of the face image. The entire protection process takes an average of 2 seconds. The similarity between 〖Face〗_adv and the original image is tested using mainstream face recognition APIs on the market. Experiments show that the average similarity between 〖Face〗_adv generated by the target model based on this application embodiment and the original image drops from 100 to 35, far below the normal threshold of face recognition models.

[0151] In some embodiments, to verify that the target adversarial sample after privacy protection processing in this application embodiment possesses a certain degree of non-recognition capability compared to the first face image before processing, this application embodiment can also conduct experimental verification. Specifically, after the similarity between the obtained target adversarial sample and the first face image is lower than a first threshold, the method further includes:

[0152] The target adversarial sample is input into the test model to obtain the test confidence. The test model is a face recognition model deployed in a simulated search engine.

[0153] If the test similarity is higher than the second threshold, then the absolute value of the first threshold is increased.

[0154] For example, experimental data is shown in Table 1 below:

[0155]

[0156] Table 1

[0157] As shown in Table 1 above, taking the target user with ID 20xxxxxxxxx534 as an example, after the target user's facial image is processed for privacy protection using the technical solution of this application embodiment, the similarity rate after being recognized by a similar facial recognition model is 1.48%. The search engine deploying this similar facial recognition model cannot match the facial image of the target user with ID 20xxxxxxxxx534 from the crawled image set using machine vision. However, the initial facial image of the target user, without any privacy protection processing, has a 100% similarity rate after being recognized by a similar facial recognition model, meaning that the facial recognition model can directly recognize it. In other words, the search engine deploying this similar facial recognition model can directly match the facial image of the target user with ID 20xxxxxxxxx534 from the crawled image set.

[0158] In some implementations, when this solution is applied to the anti-search domain, after obtaining the target adversarial sample, the business server can publish the target adversarial sample or update the first face image with the target adversarial sample.

[0159] The publication of the target adversarial sample can refer to posting moods on a social media platform. For example, a target user posts a Weibo post (a) on Weibo, where the post includes a facial image (a) of the target user and / or a specific user. This facial image (a) is a target adversarial sample obtained after privacy protection processing of the initial facial image (a') uploaded by user a. In other words, the facial image (a') is input into the target model, and the target model outputs the adversarial sample as the aforementioned facial image (a), i.e., replacing facial image (a) with facial image (a'). For example, Figure 4a For based on Figure 3 The diagram shown illustrates how a target model generates adversarial examples.

[0160] Updating the first face image to the target adversarial sample can refer to the following two scenarios (1) and (2):

[0161] (1) After processing the face images that the target user has posted on social media platforms in the past for privacy protection, replace them with the target adversarial sample.

[0162] For example, user A uploads a Weibo post (b) on the Weibo platform. This post (b) includes a facial image (b) of a target user and / or a specific user. This facial image (b) is a target adversarial sample obtained after privacy protection processing of the user's previously uploaded facial images (b'). In other words, the facial image (b') is input into a target model, and the target model's output adversarial sample is published as the aforementioned facial image (b), i.e., facial image (b) is replaced by facial image (b'). A schematic diagram can be referenced. Figure 4c , Figure 4d I will not elaborate further.

[0163] In some implementations, such as Figure 4c The attached Weibo post consists of a nine-grid image. Two of the images in the nine-grid image are partial facial images of the target user. When the business server performs privacy protection processing on the nine-grid Weibo post containing the target user, it first locates the partial facial images containing the target user (or a specific user) from the nine-grid Weibo post. Then, it inputs these partial facial images into the target model to obtain the target adversarial sample, and then replaces the nine-grid Weibo post.

[0164] (2) After processing the face images uploaded by the target user to the financial platform in the past for privacy protection, replace them with the target adversarial sample.

[0165] For example, user A uploads an ID card image c on a financial platform. This ID card image c includes the ID card images c of the target user and / or a specific user. This ID card image c is a target adversarial sample obtained after privacy protection processing of the user's historically uploaded ID card images c'. That is, the ID card image c' is input into the target model, and the target adversarial sample output by the target model is published as the aforementioned ID card image c, that is, the ID card image c' is replaced by the ID card image c.

[0166] In this embodiment, candidate adversarial samples obtained by adding perturbations to the first face image based on the target model are iterated N times. By continuously adjusting the perturbations in the candidate adversarial samples in each iteration until the loss function converges, it indicates that the target adversarial sample obtained after processing the first face image by the target model is different from the first face image before processing by the target model. Thus, the final target adversarial sample can be used as a substitute for the first face image. Therefore, on one hand, by minimizing the first similarity, the similarity between the real face and the average face can be improved. This makes the recognition result of similar face recognition models (such as models deployed in search engines) more likely to be "a common average face". The higher the similarity with the average face, the less likely it is to identify the real identity of the target user, that is, it is impossible to conclude that the real user corresponding to the target adversarial sample is the target user. On the other hand, maximizing the second and third similarities can reduce the actual similarity of the output, so that similar face recognition models (such as those deployed in search engines) tend to have lower similarity results for the adversarial sample, meaning that it is impossible to conclude that the real user corresponding to the adversarial sample is the target user.

[0167] Therefore, on the one hand, once the adversarial sample is published on a business server (such as a social platform), even if a search engine crawls the adversarial sample, it will misidentify it due to its adversarial nature, or be unable to bind it to the identity information of the target user being searched. This effectively protects the target user's privacy data from malicious collection or exploitation. On the other hand, once the first face image uploaded to the business server (such as social platforms, financial platforms, government platforms, etc., which require uploading privacy images including the target user's face) is updated with the adversarial sample, even if the adversarial sample uploaded to the business server is leaked for some reason, or is crawled by a search engine after being leaked, it will misidentify the adversarial sample due to its adversarial nature, or be unable to bind it to the identity information of the target user being searched. This effectively protects the target user's privacy data from malicious collection or exploitation.

[0168] Furthermore, because the third similarity is maximized, this adversarial example does not cause visual interference for the user; to the user, it appears as the same image. However, from a machine vision perspective, this adversarial example cannot be directly identified by an intelligent search engine based on an image recognition model and cannot be linked to the target user's identity information. Therefore, the embodiments of this application can effectively protect the privacy data of the target user.

[0169] Based on the above explanation of the source of the first face image, the following describes several methods and timing for obtaining the first face image to be used to generate adversarial examples. For example, methods for obtaining the first face image to be processed include: the target user actively initiating a privacy protection request, and the business server actively initiating a privacy protection operation for the user (e.g., the target user, a specific user).

[0170] (1) Processing scenario where the business server publishes information to a single user in real time based on event triggering.

[0171] Specifically, the user terminal sends a first message to the business server, and the business server receives the first message from the user terminal. The first message includes specific event information of the target user to be published to a social platform, or to be uploaded to a financial platform or government platform. The specific event information includes at least one initial facial image.

[0172] The business server sets the initial face image that meets the preset privacy protection conditions in the specific event information as the first face image.

[0173] In some implementations, the preset privacy protection conditions may include at least one of the following:

[0174] Including the target user's face;

[0175] Faces of specific users outside the target users;

[0176] Alternatively, it could include the face of the target user, or the face of a specific user other than the target user.

[0177] Among them, a specific user can be a user designated by the target user, or a user who has a relationship with the target user (such as an online / offline social friend relationship). This application embodiment does not limit this.

[0178] For example, the specific event information to be published could be a mood written by user A on a social media platform, such as a Weibo post or a public account article. This mood could include not only text but also images uploaded by user A, including user A's face and the face of user B. User B could be anyone other than user A, such as user A's friends, family, or public figures; this application embodiment does not limit this.

[0179] As can be seen, the business server automatically identifies the first face image that meets the preset privacy protection conditions from the specific time information to be published by the target user. Since the preset privacy protection conditions are pre-set filtering conditions, the first face image can be automatically selected so that the business server can quickly and specifically generate adversarial face samples for the target user without the target user actively triggering the process of adding perturbation patterns. Therefore, the privacy and security of the target user can be guaranteed without the target user's awareness.

[0180] (2) The business server proactively processes the historical published information of individual users in batches on a regular basis.

[0181] In some implementations, the following steps may be included:

[0182] The business server identifies target users who meet preset privacy protection conditions and obtains a set of historical images that the target user has published on social media platforms, or uploaded to financial or government platforms within a historical period. This set of historical images includes at least one unprocessed candidate historical image of the target user.

[0183] The service server sets the candidate historical images that meet the preset privacy protection conditions as the first face image.

[0184] As can be seen, the business server automatically identifies candidate historical images that meet the preset privacy protection conditions from the set of historical images posted by the target user. Since the preset privacy protection conditions are pre-set filtering conditions, on the one hand, the first face image can be automatically selected so that the business server can quickly and specifically generate adversarial face samples for the target user without the target user actively triggering the process of adding perturbation patterns. Therefore, the privacy and security of the target user can be guaranteed without the target user's awareness.

[0185] On the other hand, even if the target user does not actively request privacy protection processing (e.g., generating corresponding adversarial samples) from the business server to protect privacy when publishing any image from the aforementioned historical image set; or even if the business server does not automatically perform privacy protection processing (e.g., generating corresponding adversarial samples) to protect the target user's privacy when publishing any image from the aforementioned historical image set; or even if the business server does not periodically or irregularly perform privacy protection processing (e.g., generating corresponding adversarial samples) to protect the target user's privacy for a period of time after the target user publishes any image from the aforementioned historical image set, this solution allows for subsequent privacy protection processing (e.g., generating corresponding adversarial samples) on candidate historical images in the historical image set that meet the preset privacy protection conditions. In other words, it can improve the target user's need for privacy security by identifying and addressing omissions or providing compensation.

[0186] (3) Scenarios where the business server processes real-time / historical information published by a single user based on user-initiated requests.

[0187] In some implementations, the following steps may be included:

[0188] The user terminal sends a second message to the business server, and the business server receives the second message from the user terminal. The second message instructs that at least one facial image from a specific event information item to be published to a social media platform, or uploaded to a financial or government platform, undergo privacy protection processing.

[0189] According to the instructions of the second message, the business server sets the face image that meets the preset privacy protection conditions in the specific event information as the first face image.

[0190] It is evident that by having the target user proactively request privacy protection processing (such as generating corresponding adversarial samples) from the business server when publishing any image from the aforementioned historical image set, the privacy and security of the target user can be protected.

[0191] (4) The business server updates the mapping relationship between input and output.

[0192] In some implementations, the following steps may be included:

[0193] The business server determines the number of inputs to the target model, and according to the number of inputs, inputs each sub-image in the first face image as an input to the target model to obtain the target adversarial sample, wherein each input corresponds to a pixel region in the first face image.

[0194] Alternatively, the business server determines the number of inputs to the target model, and according to the number of inputs, inputs each of the first face images as an input to the target model to obtain the target adversarial samples corresponding to each first face image.

[0195] Each input corresponds to a pixel region in the first face image.

[0196] As can be seen, by updating the mapping relationship between input and output, the business server can better adapt to image inputs containing different pixel content and perform targeted processing, thereby improving processing efficiency.

[0197] The above three methods for obtaining the first face image of the face to be generated adversarial sample can be flexibly switched, configured, or combined, and any one or at least two of them can be implemented. This application embodiment does not limit this.

[0198] See Figure 5 , Figure 5 This paper illustrates a flowchart of a face image acquisition method according to an embodiment of this application. The following describes a face image acquisition method provided by this application embodiment, which is applied to a search engine. This application embodiment includes:

[0199] 501. The search engine receives the search command.

[0200] The search instructions include key information about the target user, such as the target user's name and nickname. The target user can be at least one user to be searched, and the search engine can simultaneously search for images related to at least two target users; however, this embodiment does not limit this.

[0201] 502. In response to the search command, the search engine sends an access request to the business server.

[0202] This access request is used to request a facial image that matches key information from the business server. For example, if the search instruction includes the name Zhang Wenwen, the access request could be to request a facial image tagged "Zhang Wenwen" from platforms such as Weibo and various public image libraries.

[0203] 503. The search engine obtains a set of candidate images that match the key information from the business server.

[0204] The candidate image set includes at least one target adversarial sample, which is obtained by inputting a first face image into a target model; the target model is obtained by unsupervised learning based on unsupervised learning, and the similarity between the target adversarial sample and the first face image is lower than a preset threshold.

[0205] 504. The search engine determines the target image set from the candidate image set as the search results for the target user.

[0206] The target image set does not include adversarial examples of faces. This target image set refers to images containing at least one face that have been posted by at least one target user. This target image set can originate from face images of the target user posted by user a, or it can originate from face images of the target user posted by user b; this embodiment of the application does not limit this.

[0207] Specifically, the candidate image set can be input into the face recognition model deployed in the search engine to obtain the first similarity between the target adversarial sample and the second face image, and the second similarity between each first face image and the second face image; the first similarity is lower than the first threshold, the second similarity is not less than the third threshold, and the first threshold is less than the third threshold;

[0208] The first face image selected from the candidate image set, whose similarity to the second face image is not less than the third threshold, is selected as the target image set.

[0209] As can be seen, in this embodiment, because the business server performs privacy protection processing on at least one image including a face published by the user, at least one image including a face generates a target adversarial sample. Therefore, when other search tools (such as the search engine in this embodiment) search and crawl these candidate image sets from the business server according to search instructions including the aforementioned key information, even if the search engine crawls a target adversarial sample containing the target user, it cannot intelligently identify the target adversarial sample matching the key information of the target user when preprocessing (e.g., filtering) the candidate image set. That is, the target image set identified by the search engine from the candidate image set will not include the aforementioned target adversarial sample. Thus, on the one hand, this embodiment can interfere with face recognition by generating target adversarial samples, thereby resisting the search engine based on the neural network model from filtering out face images containing the target user (i.e., the aforementioned target adversarial samples). On the other hand, it also effectively protects the privacy and security of user data.

[0210] Figures 1 to 6Any technical feature mentioned in the embodiments corresponding to any one of the above also applies to the embodiments of this application. Figures 7 to 11 The corresponding implementation examples will not be repeated hereafter.

[0211] The above describes a face image processing method according to an embodiment of this application. The following describes the image processing device and the image search device that perform the above face image processing method.

[0212] See Figure 7 ,like Figure 7 The schematic diagram of an image processing device 40 shown illustrates its ability to perform privacy protection processing on images to be published and previously published images, thereby generating adversarial examples of faces. This prevents a search engine based on a neural network model from directly identifying the first face image matching the specific user even after crawling face images of a specific user from the web (e.g., crawling face images previously published by the target user from a business server) and preprocessing the face images. The image processing device 40 in this embodiment can achieve the above-described... Figures 1-6 The steps in the face image processing method executed by the image processing device 40 in any corresponding embodiment. The functions implemented by the image processing device 40 can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. The image processing device 40 may include an input / output module 401 and a processing module 402. The functional implementation of the input / output module 401 and the processing module 402 can be found in [reference needed]. Figures 1-6 The operations performed in any of the corresponding embodiments will not be described in detail here.

[0213] In some implementations, the input / output module 401 can be used to acquire a first face image to be processed;

[0214] The processing module 402 can be used to extract the original facial features and the intermediate layer original features of the target model from the first facial image;

[0215] Add a target perturbation to the first face image to obtain candidate adversarial examples;

[0216] The candidate adversarial samples are input into the target model to extract face adversarial features and intermediate layer adversarial features;

[0217] Calculate the first similarity between the preset average face feature and the adversarial face feature, calculate the second similarity between the original face feature and the adversarial face feature, and calculate the third similarity between the original intermediate layer feature and the adversarial intermediate layer feature;

[0218] If the first similarity is less than the first preset threshold, the second similarity is greater than the second preset threshold, and the third similarity is greater than the third preset threshold, then the candidate adversarial sample is back-derived to obtain the target gradient.

[0219] The first similarity is minimized, the second similarity and the third similarity are maximized, the perturbation in the candidate adversarial sample is updated according to the target gradient, and the updated target adversarial sample is input into the target model as the candidate adversarial sample until the similarity between the obtained target adversarial sample and the first face image is lower than the first threshold.

[0220] In some implementations, the input / output module 401 is further configured to publish the target adversarial sample obtained by the processing module 402, or the processing module 402 updates the first face image with the target adversarial sample.

[0221] In some embodiments, the processing module 402 is specifically used for:

[0222] Update the pixel value of each pixel in the candidate adversarial sample according to the target gradient.

[0223] In some embodiments, the processing module 402 is further configured to:

[0224] If the first similarity is greater than the first preset threshold, the second similarity is less than the second preset threshold, and the third similarity is less than the second preset threshold, then the candidate adversarial sample is taken as the target adversarial sample.

[0225] In some implementations, after the processing module 402 obtains a similarity between the target adversarial sample and the first face image that is lower than a first threshold, it is further configured to:

[0226] The target adversarial sample is input into the test model to obtain the test confidence. The test model is a face recognition model deployed in a simulated search engine.

[0227] If the test similarity is higher than the second threshold, then the absolute value of the first threshold is increased.

[0228] In some embodiments, the processing module 402 is specifically used for:

[0229] The input / output module 401 receives a first message from the user terminal. The first message includes specific event information that the target user is to publish to a social platform, or to upload to a financial platform or a government platform. The specific event information includes at least one initial face image.

[0230] The initial face image that meets the preset privacy protection conditions in the specific event information is set as the first face image.

[0231] In some embodiments, the processing module 402 is specifically used for:

[0232] Identify target users who meet the preset privacy protection conditions;

[0233] Obtain a set of historical images that the target user has published to social media platforms or uploaded to financial or government platforms during a historical period. The set of historical images includes at least one unprocessed candidate historical image of the target user.

[0234] The candidate historical image that meets the preset privacy protection conditions is set as the first face image.

[0235] In some implementations, the first face image is derived from at least one of the following methods:

[0236] The target user has specific event information to be posted on a social media platform, and the specific event information includes at least one initial facial image;

[0237] Alternatively, a set of historical images posted by the target user on a social media platform within a historical period, the set of historical images including at least one unprocessed candidate historical image of the target user, the candidate historical images including images of the target user that meet preset privacy protection conditions.

[0238] In some implementations, the first face image is derived from at least one of the following methods:

[0239] The target user's historical image set to be uploaded to financial or government platforms;

[0240] Alternatively, the target user's historical image set uploaded to financial or government platforms during a historical period;

[0241] The historical image set includes at least one unprocessed candidate historical image of the target user.

[0242] In some embodiments, the processing module 402 is specifically used for:

[0243] The input / output module 401 receives a second message from the user terminal, the second message instructing that at least one facial image in the specific event information to be published to a social platform or uploaded to a financial platform or government platform by the target user be subjected to privacy protection processing.

[0244] According to the instructions of the second message, the face image that meets the preset privacy protection conditions in the specific event information is set as the first face image.

[0245] In some embodiments, the processing module 402 is specifically used for:

[0246] Determine the number of inputs to the target model;

[0247] According to the number of inputs, each sub-image in the first face image is input to the target model as an input to obtain the target adversarial sample, wherein each input corresponds to a pixel region in the first face image.

[0248] In this embodiment, candidate adversarial samples obtained by adding perturbations to the first face image based on the target model are iterated N times. By continuously adjusting the perturbations in the candidate adversarial samples in each iteration until the loss function converges, it indicates that the target adversarial sample obtained after processing the first face image by the target model is different from the first face image before processing by the target model. Thus, the final target adversarial sample can be used as a substitute for the first face image. Therefore, on one hand, by minimizing the first similarity, the similarity between the real face and the average face can be improved. This makes the recognition result of similar face recognition models (such as models deployed in search engines) more likely to be "a common average face". The higher the similarity with the average face, the less likely it is to identify the real identity of the target user, that is, it is impossible to conclude that the real user corresponding to the target adversarial sample is the target user. On the other hand, maximizing the second and third similarities can reduce the actual similarity of the output, so that similar face recognition models (such as those deployed in search engines) tend to have lower similarity results for the adversarial sample, meaning that it is impossible to conclude that the real user corresponding to the adversarial sample is the target user.

[0249] Therefore, on the one hand, once the adversarial sample is published on a business server (such as a social platform), even if a search engine crawls the adversarial sample, it will misidentify it due to its adversarial nature, or be unable to bind it to the identity information of the target user being searched. This effectively protects the target user's privacy data from malicious collection or exploitation. On the other hand, once the first face image uploaded to the business server (such as social platforms, financial platforms, government platforms, etc., which require uploading privacy images including the target user's face) is updated with the adversarial sample, even if the adversarial sample uploaded to the business server is leaked for some reason, or is crawled by a search engine after being leaked, it will misidentify the adversarial sample due to its adversarial nature, or be unable to bind it to the identity information of the target user being searched. This effectively protects the target user's privacy data from malicious collection or exploitation.

[0250] The image processing device 40, which performs the face image processing method, and the image search device 50, which performs the face image acquisition method, in the embodiments of this application have been described above from the perspective of modular functional entities. The image processing device 40 and the image search device 50, which perform the face image processing method, in the embodiments of this application will be described below from the perspective of hardware processing. It should be noted that in the embodiments of this application… Figure 7 In the illustrated embodiment, the physical device corresponding to the input / output module 401 can be an input / output unit, transceiver, radio frequency circuit, communication module, and output interface, etc., and the physical device corresponding to the processing module 402 can be a processor. Figure 7 The image processing device 40 shown may have, for example: Figure 9 The structure shown, when Figure 7 The image processing device 40 shown has, for example Figure 9 When the structure shown is used, Figure 9 The processor and transceiver in the image processing device 40 can perform the same or similar functions as the input / output module 401 and processing module 402 provided in the aforementioned device embodiment corresponding to the image processing device 40. Figure 9 The memory stores the computer programs that the processor needs to call when executing the above face image processing method.

[0251] In the embodiments of this application Figure 8 In the embodiments shown, the physical device corresponding to the transceiver module 501 can be an input / output unit, transceiver, radio frequency circuit, communication module, and output interface, etc., and the physical device corresponding to the processing module 502 can be a processor. Figure 8 The image search device 50 shown can have, for example, Figure 9 The structure shown, when Figure 8 The image search device 50 shown has, for example: Figure 9 When the structure shown is used, Figure 9 The processor and transceiver in the device can perform the same or similar functions as the transceiver module 501 and processing module 502 provided in the aforementioned device embodiment corresponding to the image search device 40. Figure 9 The memory stores the computer programs that the processor needs to call when executing the above face image acquisition method.

[0252] This application also provides another image processing apparatus, such as... Figure 10As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The image processing device can be any image processing device, including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) image processing devices, in-vehicle computers, etc. Taking a mobile phone as an example:

[0253] Figure 10 This is a block diagram showing a portion of the structure of a mobile phone related to the image processing apparatus provided in the embodiments of this application. (Reference) Figure 10 The mobile phone includes: a radio frequency (RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 780, an audio circuit 760, a wireless-fidelity (Wi-Fi) module 7100, a processor 780, and a power supply 790, among other components. Those skilled in the art will understand that... Figure 7 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0254] The following is combined Figure 10 A detailed introduction to each component of a mobile phone:

[0255] The RF circuit 710 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 780; additionally, it transmits uplink data to the base station. Typically, the RF circuit 710 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 710 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0256] The memory 720 can be used to store software programs and modules. The processor 780 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 720. The memory 720 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0257] The input unit 730 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732. The touch panel 731, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 731), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 731 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 780, and can also receive and execute commands sent by the processor 780. In addition, the touch panel 731 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 731, the input unit 730 may also include other input devices 732. Specifically, other input devices 732 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0258] The display unit 740 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 740 may include a display panel 741, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 741. Further, a touch panel 731 may cover the display panel 741. When the touch panel 731 detects a touch operation on or near it, it transmits the information to the processor 780 to determine the type of touch event. Subsequently, the processor 780 provides corresponding visual output on the display panel 741 based on the type of touch event. Although in Figure 7 In this embodiment, the touch panel 731 and the display panel 741 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 731 and the display panel 741 can be integrated to realize the input and output functions of the mobile phone.

[0259] The mobile phone may also include at least one sensor 780, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 741 according to the ambient light level, and the proximity sensor can turn off the display panel 741 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0260] Audio circuit 760, speaker 761, and microphone 762 provide an audio interface between the user and the mobile phone. Audio circuit 760 converts received audio data into electrical signals and transmits them to speaker 761, where speaker 761 converts them into sound signals for output. On the other hand, microphone 762 converts collected sound signals into electrical signals, which are received by audio circuit 760, converted into audio data, and then processed by processor 780 before being transmitted via RF circuit 710 to, for example, another mobile phone, or the audio data can be output to memory 720 for further processing.

[0261] Wi-Fi is a short-range wireless transmission technology. Through the Wi-Fi module 7100, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 9 The Wi-Fi module 7100 is shown, but it is understood that it is not a necessary component of the mobile phone and can be omitted as needed without changing the nature of the application.

[0262] The processor 780 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 720, and calls data stored in the memory 720 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 780 may include one or more processing units; preferably, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 780.

[0263] The mobile phone also includes a power supply 790 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 780 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0264] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0265] In this embodiment of the application, the processor 780 included in the mobile phone also has the function of controlling and executing the above-mentioned... Figure 10 The method flow executed by the image processing apparatus 40 shown above. The steps executed by the image processing apparatus in the above embodiments can be based on this... Figure 10 The mobile phone structure is shown. For example, the processor 722 performs the following operations by calling instructions from memory 732:

[0266] The first face image to be processed is obtained through the input unit 730;

[0267] The first face image acquired by the input unit 730 is input into the target model to obtain a target adversarial sample. The target model is obtained by unsupervised learning based on unsupervised learning, and the similarity between the target adversarial sample and the first face image is lower than a preset threshold.

[0268] The processing module publishes the target adversarial sample obtained by the processing module through the input unit 730, or the processing module updates the first face image with the target adversarial sample.

[0269] In other embodiments of this application, the processor 780 included in the mobile phone also has the function of controlling the execution of the above-mentioned components. Figure 8 The method flow executed by the image search device 50 shown is illustrated. The steps performed by the image processing device in the above embodiments can be based on this... Figure 10 The mobile phone structure is shown. For example, the processor 722 performs the following operations by calling instructions from memory 732:

[0270] The input unit 730 receives a search instruction, which includes key information about the target user.

[0271] In response to the search command, the transceiver module obtains a set of candidate images matching the key information from the business server; wherein, the set of candidate images includes at least one adversarial sample, the adversarial sample is obtained by inputting a first face image into a target model; the target model is obtained by unsupervised learning based on unsupervised learning, and the similarity between the adversarial sample and the first face image is lower than a preset threshold;

[0272] The target image set is determined from the candidate image set as the search results for the target user, and the target image set does not include target adversarial samples.

[0273] This application also provides another image processing apparatus for implementing the above-described face image processing method, or a search apparatus for implementing the above-described face image acquisition method, such as... Figure 11 As shown, Figure 11 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1020 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1022 (e.g., one or more processors) and memory 1032, and one or more storage media 1030 (e.g., one or more mass storage devices) for storing application programs 1042 or data 1044. The memory 1032 and storage media 1030 can be temporary or persistent storage. The program stored in the storage media 1030 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 1022 may be configured to communicate with the storage media 1030 and execute the series of instruction operations in the storage media 1030 on the server 1020.

[0274] Server 1020 may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input / output interfaces 1058, and / or one or more operating systems 1041, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0275] In the above embodiments, the service server (e.g.) Figure 7 The image processing device 40 shown) or search engine (e.g. Figure 8 The steps performed by the search device 50 shown can be based on this Figure 11 The structure of server 1020 is shown. For example, in the above embodiment, it consists of... Figure 7 The steps performed by the image processing device 40 shown can be based on this Figure 11 The server structure shown.

[0276] For example, the processor 1022 performs the following operations by calling instructions stored in memory 1032:

[0277] The first face image to be processed is obtained through the input / output interface 1058;

[0278] Extract the original facial features and the intermediate layer original features of the target model from the first facial image obtained from the input / output interface 1058;

[0279] Add a target perturbation to the first face image to obtain candidate adversarial examples;

[0280] The candidate adversarial samples are input into the target model to extract face adversarial features and intermediate layer adversarial features;

[0281] Calculate the first similarity between the preset average face feature and the adversarial face feature, calculate the second similarity between the original face feature and the adversarial face feature, and calculate the third similarity between the original intermediate layer feature and the adversarial intermediate layer feature;

[0282] If the first similarity is less than the first preset threshold, the second similarity is greater than the second preset threshold, and the third similarity is greater than the third preset threshold, then the candidate adversarial sample is back-derived to obtain the target gradient.

[0283] The first similarity is minimized, the second similarity and the third similarity are maximized, the perturbation in the candidate adversarial sample is updated according to the target gradient, and the updated target adversarial sample is input into the target model as the candidate adversarial sample until the similarity between the obtained target adversarial sample and the first face image is lower than the first threshold.

[0284] In some implementations, the target adversarial sample obtained by the processing module can also be published through the input / output interface 1058, or the processing module can update the first face image with the target adversarial sample.

[0285] For example, in the above embodiments, by Figure 8 The steps performed by the search device 50 shown can be based on this Figure 11 The server architecture is shown. For example, the processor 1022 performs the following operations by calling instructions from memory 1032:

[0286] The search command is received through the input / output interface 1058, and the search command includes key information of the target user.

[0287] In response to the search command, the transceiver module obtains a set of candidate images matching the key information from the business server; wherein, the set of candidate images includes at least one first face image and at least one adversarial sample, the adversarial sample being obtained by inputting the first face image into a target model; the similarity between the adversarial sample and the first face image is lower than a preset threshold;

[0288] The target image set is determined from the candidate image set as the search results for the target user, and the target image set does not include target adversarial samples.

[0289] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0290] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0291] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0292] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0293] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0294] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0295] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0296] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A face image processing method, characterized by, The method is executed by a business server, and the method comprises: obtaining a first face image to be processed; inputting the first face image into a target model to extract face original features and intermediate layer original features of the target model from the first face image to be processed; wherein the target model is trained based on an average face image; the average face image refers to a composite face image obtained according to an average value of measurement data; adding a target disturbance to the first face image to obtain a candidate adversarial sample; inputting the candidate adversarial sample into the target model to extract face adversarial features and intermediate layer adversarial features; calculating a first similarity between preset average face features and the face adversarial features, a second similarity between the face original features and the face adversarial features, and a third similarity between the intermediate layer original features and the intermediate layer adversarial features; if the first similarity is less than a first preset threshold, the second similarity is greater than a second preset threshold, and the third similarity is greater than a third preset threshold, then performing back propagation on the candidate adversarial sample to obtain a target gradient; performing minimization processing on the first similarity, maximization processing on the second similarity and the third similarity, updating the disturbance in the candidate adversarial sample according to the target gradient, inputting the target adversarial sample obtained by the updating into the target model as the candidate adversarial sample, and repeating the above operations until the similarity between the target adversarial sample and the first face image is lower than a first threshold; publishing the target adversarial sample or updating the first face image to the target adversarial sample.

2. The method of claim 1, wherein, After the back propagation on the candidate adversarial sample to obtain the target gradient, before the updating of the pixel values of each pixel point in the candidate adversarial sample according to the target gradient, the method further comprises: performing maximization processing on a third loss function.

3. The method of claim 1, wherein, After the similarity between the target adversarial sample and the first face image is lower than the first threshold, the method further comprises: inputting the target adversarial sample into a test model to obtain a test confidence, wherein the test model is a face recognition model deployed in a search engine; if the test confidence is higher than a second threshold, then expanding the absolute value of the first threshold.

4. The method of claim 3, wherein, The obtaining of the first face image to be processed comprises: receiving a first message from a user terminal, wherein the first message comprises specific event information to be published to a social platform or uploaded to a financial platform or a government platform by a target user, and the specific event information comprises at least one initial face image; setting the initial face image in the specific event information that meets a preset privacy protection condition as the first face image.

5. The method of claim 3, wherein, The obtaining of the first face image to be processed comprises: determining a target user who meets a preset privacy protection condition; obtaining a historical image set of the target user that has been published to a social platform or uploaded to a financial platform or a government platform in a historical period, wherein the historical image set comprises at least one candidate historical image of the target user that has not been preprocessed. The candidate historical image meeting the preset privacy protection condition is set as the first face image.

6. The method of claim 3, wherein, The first face image to be processed is obtained, including: receiving a second message from the user terminal, the second message indicating that at least one face image in specific event information to be published by a target user to a social platform or to be uploaded to a financial platform or a government affairs platform is to be subjected to privacy protection processing; According to the indication of the second message, the face image in the specific event information that meets the preset privacy protection condition is set as the first face image.

7. A processing device, characterized by The computer readable storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by the processor to execute the method of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by the processor to execute the method of any one of claims 1-6.

Images