Model training method and device, identity anonymization method and device, equipment, storage medium, and computer program

The model training method enhances identity anonymization by projecting images into a target space to generate virtual identities, addressing quality deterioration issues in existing GAN-based methods, and producing high-resolution, artifact-free anonymized images.

JP7577131B2Active Publication Date: 2024-11-01TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2022566254
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2022-03-10
Filing Date
2022-08-11
Publication Date
2024-11-01
Estimated Expiration
2042-08-11

AI Technical Summary

Technical Problem

Existing methods for identity anonymization in images using conditional generative adversarial networks (GANs) often result in deteriorated image quality due to the removal of facial regions.

Method used

A model training method that projects training images into a target space to obtain virtual identity vectors and extracts attribute vectors, using a fusion module to generate identity-anonymized images without removing facial regions, ensuring high fidelity and resolution.

Benefits of technology

The method improves the quality of identity-anonymized images by generating images with virtual identities while preserving attribute information, resulting in high-resolution and artifact-free anonymized images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577131000027
    Figure 0007577131000027
  • Figure 0007577131000028
    Figure 0007577131000028
  • Figure 0007577131000029
    Figure 0007577131000029
Patent Text Reader

Abstract

The present application provides a model training method and device, an identity anonymization method and device, a device, a storage medium and a program product, which are applicable to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, driving assistance, etc. The method includes: sampling in a target space of a projection module in a target network model to obtain N virtual identity vectors, performing attribute vector extraction on a processing target image by an attribute module in the target network model to obtain M attribute vectors, and performing image generation according to the N virtual identity vectors and the M attribute vectors by a fusion module in the target network model to obtain an identity anonymized image of the processing target image.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is filed based on a Chinese patent application bearing application number 202210234385.2, filed with the China Patent Office on March 10, 2022, and claims priority to the Chinese patent application, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of image processing, and in particular to a model training method and apparatus, an identity anonymization method and apparatus, an apparatus, a storage medium, and a program product. [Background technology]

[0003] Identity anonymization, also known as de-identification, is the removal of identifiable identity features in an image or video while leaving other identity-irrelevant attributes unchanged, ensuring that the anonymized photo or video is still visually realistic.

[0004] In the related technology, anonymized photos are generated using conditional generative adversarial networks (GANs), pose keypoints of the original photos are extracted, and the pose keypoints of the original photos and the background photos after removing the face area are input as conditions to a model, and a new virtual identity is generated to fill the empty face area. However, since this method inputs the background photos after removing the face area as a model, the image quality generated by the model is poor. Summary of the Invention [Problem to be solved by the invention]

[0005] Embodiments of the present application provide a model training method and apparatus, an identity anonymization method and apparatus, a computing device, a computer-readable storage medium, and a computer program product, which can improve the generation quality of identity-anonymized images. [Means for solving the problem]

[0006] An embodiment of the present application provides a model training method, the method comprising: Projecting the first training image into the target space by a projection module in the target network model to obtain N first virtual identity vectors, where N is a positive integer; Performing attribute vector extraction on the second training image by an attribute module in the target network model to obtain M attribute vectors (M is a positive integer); Performing image generation according to the N first virtual identity vectors and the M attribute vectors by a fusion module in the target network model to obtain an identity-anonymized image of the second training image; determining a loss for the target network model based on the identity-anonymized image; and training the target network model based on the loss.

[0007] An embodiment of the present application further provides an identity anonymization method, the method comprising: Sampling in a target space of a projection module in a target network model to obtain N virtual identity vectors (N is a positive integer); Extracting attribute vectors from the image to be processed by an attribute module in the target network model to obtain M attribute vectors (M is a positive integer); and performing image generation based on the N virtual identity vectors and the M attribute vectors by a fusion module in the target network model to obtain an identity-anonymized image of the processing target image.

[0008] An embodiment of the present application further provides a model training apparatus, the apparatus comprising: a projection unit, an attribute unit, a fusion unit, and a training unit; The projection unit is configured to project a first training image into a target space by a projection module in a target network model to obtain N first virtual identity vectors, where N is a positive integer; The attribute unit is configured to perform attribute vector extraction on the second training image through an attribute module in the target network model to obtain M attribute vectors, where M is a positive integer; The fusion unit is configured to perform image generation according to the N virtual identity vectors and the M attribute vectors through a fusion module in the target network model to obtain an identity anonymized image of the second training image; The training unit is configured to determine a loss for the target network model based on the identity-anonymized image, and train the target network model based on the loss.

[0009] An embodiment of the present application further provides an identity anonymization device, the device comprising: a sampling unit, an attribute unit, and an anonymization unit; The sampling unit is configured to sample in a target space of a projection module in a target network model to obtain N virtual identity vectors, where N is a positive integer; The attribute unit is configured to perform attribute vector extraction for the processing target image through an attribute module in a target network model, and obtain M attribute vectors (M is a positive integer); The anonymization unit is configured to perform image generation based on the N virtual identity vectors and the M attribute vectors by a fusion module in the target network model, to obtain an identity-anonymized image of the processing target image.

[0010] The present embodiment further provides a computing device including a processor and a memory, the memory storing a computer program, and the processor calling and executing the computer program stored in the memory to perform the above-mentioned model training method or the identity anonymization method according to the present embodiment.

[0011] The present embodiment further provides a chip configured to realize the above-mentioned model training method or identity anonymization method according to the embodiment of the present application, the chip comprising a processor for executing the above-mentioned model training method or identity anonymization method according to the embodiment of the present application in a device equipped with the chip by calling and executing a computer program from a memory.

[0012] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, causes a computer to implement the above-described model training method or identity anonymization method according to an embodiment of the present application.

[0013] An embodiment of the present application further provides a computer program product comprising computer program instructions, which when executed by a computer, cause the computer to implement the above-described model training method or identity anonymization method according to an embodiment of the present application.

[0014] An embodiment of the present application further provides a computer program, which, when executed on a computer, causes the computer to implement the above model training method or the identity anonymization method according to the embodiment of the present application. Effect of the Invention

[0015] The embodiments of the present application have the following beneficial effects: In the training process of the target network model, the first training image is projected into the target space to obtain N first virtual identity vectors, so that the target network model can fully learn the identity information in the image; attribute vector extraction is performed on the second training image to obtain M attribute vectors, so that the target network model can fully learn the attribute information in the image; and image generation is performed based on the N first virtual identity vectors and the M attribute vectors to obtain an identity-anonymized image of the second training image; in this way, the trained model can generate an image that carries virtual identity information without changing the attribute information of the original image.

[0016] In the application process of the target network model, the target space of the projection module is sampled to obtain N virtual identity vectors, thereby realizing the generation of virtual identity information; attribute vector extraction is performed on the image to be processed to obtain M attribute vectors, thereby preventing the loss of attribute features in the image to be processed, thereby ensuring the quality of the generated identity anonymized image; image generation is performed according to the N virtual identity vectors and the M attribute vectors to obtain the identity anonymized image of the image to be processed, thereby generating an identity anonymized image carrying virtual identity information, i.e., an identity anonymized image that hides the real identity, without changing the attribute information of the image to be processed. That is, the embodiment of the present application does not remove the face area in the image during identity anonymization, and uses the target network model to generate an independent virtual identity, thereby improving the fidelity and resolution of identity anonymization. [Brief description of the drawings]

[0017] [Figure 1A]FIG. 2 is a schematic diagram of a real image according to an embodiment of the present application. [Figure 1B] FIG. 1B is a schematic diagram of an identity anonymization image corresponding to FIG. 1A according to an embodiment of the present application. [Figure 1C] FIG. 1B is a schematic diagram of an identity anonymization image corresponding to FIG. 1A according to an embodiment of the present application. [Figure 1D] FIG. 1B is a schematic diagram of an identity anonymization image corresponding to FIG. 1A according to an embodiment of the present application. [Diagram 2] FIG. 1 is a schematic diagram of a system architecture according to an embodiment of the present application. [Diagram 3] 1 is an exemplary flowchart of a model training method according to an embodiment of the present application. [Figure 4] FIG. 2 is an exemplary structural diagram of a target network model according to an embodiment of the present application; [Diagram 5] FIG. 2 is an exemplary structural diagram of a target network model according to an embodiment of the present application; [Figure 6] FIG. 2 is an exemplary structural diagram of a target network model according to an embodiment of the present application; [Figure 7] FIG. 2 is an exemplary structural diagram of a fusion module according to an embodiment of the present application. [Figure 8] FIG. 2 is an exemplary structural diagram of a target network model according to an embodiment of the present application; [Figure 9] FIG. 2 is a schematic diagram of the determination of contrast loss according to an embodiment of the present application. [Figure 10] FIG. 2 is a schematic diagram of the determination of contrast loss according to an embodiment of the present application. [Figure 11] 1 is an exemplary flow chart of an identity anonymization method according to an embodiment of the present application; [Figure 12] FIG. 2 is a schematic diagram of a projection module according to an embodiment of the present application; [Figure 13] FIG. 2 is a schematic diagram of determining an identity anonymized image according to an embodiment of the present application. [Figure 14] FIG. 1 is an exemplary block diagram of a model training device according to an embodiment of the present application. [Figure 15]FIG. 2 is an exemplary block diagram of an identity anonymization device according to an embodiment of the present application. [Figure 16] FIG. 2 is an exemplary block diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, the technical solutions in the embodiments of the present application will be described with reference to the drawings in the embodiments of the present application. In the embodiment of the present invention, it should be understood that "B corresponding to A" means that B is associated with A. In one embodiment, B can be determined according to A. Furthermore, it should be understood that determining B according to A does not mean determining B according to A only, and B can also be determined according to A and / or other information. In describing the embodiments of the present application, unless otherwise specified, "plurality" means two or more than two.

[0019] In order to clearly describe the technical solutions of the embodiments of the present application, the embodiments of the present application use terms such as "first", "second", "third", etc. to distinguish between items that have the same or similar functions and actions. Those skilled in the art can understand that the terms "first", "second", "third", etc. are not limited to the number and execution order, and the terms "first", "second", "third", etc. are not necessarily limited to different.

[0020] In order to facilitate understanding of the embodiments of the present application, a brief description of relevant concepts of the embodiments of the present application will first be provided.

[0021] Artificial Intelligence (AI) is the theory, methods, techniques and application systems that use digital computers or machines controlled by digital computers to simulate and extend human intelligence, sense the environment, acquire knowledge, and use the knowledge to obtain optimal results. In other words, AI is a comprehensive technology of computer science that aims to understand the nature of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, equipping machines with the capabilities of sensing, reasoning and decision-making.

[0022] Artificial intelligence technology is a comprehensive academic field that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence usually include sensors, inherent artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interactive systems, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, voice processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0023] Machine learning (ML) is a multidisciplinary cross-subject that deals with subjects such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It is the specialized study of how computers simulate or realize human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continue improving their own performance. Machine learning is at the heart of artificial intelligence and is the fundamental method of making computers intelligent, and is applied to each field of artificial intelligence. Machine learning and deep learning usually include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0024] The method of the embodiment of the present application can be applied to any scenario where anonymization processing needs to be performed on an image. For example, as shown in Figure 1A to Figure 1D, Figure 1A is a real image, and Figure 1B to Figure 1D are identity-anonymized images of Figure 1A. Comparing Figure 1A with Figure 1B to Figure 1D, Figure 1B to Figure 1D removes the identifiable identity features (Identity) in Figure 1A, while at the same time not changing other attributes unrelated to identity, ensuring that it is still visually realistic.

[0025] In scenario 1, the embodiment of the present application can be applied to a privacy protection scenario, for example, for a face-related photo or video, the method of the embodiment of the present application can be used to switch the real identity to a virtual identity, so that the user can continue to perform tasks such as subsequent detection without leaking privacy. Furthermore, when publishing a photo or video, the user can also use the method of the embodiment of the present application to hide his / her identity features, to avoid leaking real information.

[0026] In scenario 2, the embodiments of the present application can be applied to scenarios of generating virtual images, for example, the technical solutions of the embodiments of the present application can be used to generate virtual identities, for example, fixing identity hidden variables, replacing background photos, and generating photos or videos of a certain virtual image in different scenarios.

[0027] It should be noted that the above scenarios 1 and 2 are described using an example in which the target is a face, and the method of the embodiments of the present application can also be applied to scenarios of identity anonymization of targets other than faces, for example, identity anonymization can be performed on any target, such as an animal or a vehicle, in the image to be processed.

[0028] In some embodiments, the method of the embodiments of the present application may be applied to an intelligent traffic system, and the intelligent traffic system (ITS), also called the intelligent transportation system, effectively and comprehensively applies advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operation research, artificial intelligence, etc.) to traffic transportation, service control and vehicle manufacturing, and strengthens the relationship between vehicles, roads and users to form an integrated transportation system that ensures safety, improves efficiency, improves the environment and saves energy. For example, the technical solution of the present application combined with intelligent traffic may be that an in-vehicle device collects a user's face image, and performs identity anonymization processing on the collected face image according to the method of the embodiments of the present application, and then transmits it to another device to perform task analysis such as illegal driving analysis or intelligent driving analysis.

[0029] FIG. 2 is a schematic diagram of a system architecture according to an embodiment of the present application, the system comprising a user device 101, a data collection device 102, a training device 103, an execution device 104, a database 105, a content library 106, an I / O interface 107 and a target network model 108.

[0030] Here, the data collection device 102 is configured to read training data from the content library 106 and store the read training data in the database 105. The training data according to the embodiment of the present application includes a first training image, a second training image, and a third training image, and the first training image, the second training image, and the third training image are all used to train a target network model.

[0031] In some embodiments, the user device 101 is configured to perform annotation operations on the data in the database 105 .

[0032] The training device 103 trains the target network model 108 based on the training data held in the database 105, such that the trained target network model 108 is capable of generating identity-anonymized images of the images to be processed. In some embodiments, the target network model 108 obtained by the training device 103 can be applied to different systems or devices.

[0033] 2, the execution device 104 is provided with an I / O interface 107 for exchanging data with external devices. For example, the execution device 104 receives a processing target image sent by the user device 101 through the I / O interface. The computing module 109 of the execution device 104 uses the trained target network model 108 to process the input processing target image, output an identity anonymized image, and output the generated identity anonymized image to the user device 101 for display, or input it to other task models to perform other task processing.

[0034] Here, the user equipment 101 may include a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a mobile internet device (MID), or other terminal equipment having a function of installing a browser.

[0035] The execution device 104 may be a server. The server 1 may be one or more. When there are multiple servers, at least one of a case where at least two servers are configured to provide different services and a case where at least two servers are configured to provide the same service may exist, for example, the same service may be provided by a load balancing method, and the embodiment of the present application is not limited thereto. Here, the above server may be an independent physical server, a server cluster or a distributed system configured by multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data, and artificial intelligence platforms. The server may be a node of the blockchain.

[0036] In this embodiment, the execution device 104 is connected to the user device 101 through a network, which may be a wireless or wired network, such as an Intranet, the Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, a Bluetooth network, a Wi-Fi network, a telephone network, etc.

[0037] 2 is merely a schematic diagram of a system architecture according to an embodiment of the present application, and the positional relationships between the equipment, devices, modules, etc. shown in the figure are not intended to be limiting. In some embodiments, the data collection equipment 102 may be the same equipment as the user equipment 101, the training equipment 103, and the execution equipment 104. The database 105 may be distributed in one server or multiple servers, and the content library 106 may be distributed in one server or multiple servers.

[0038] The following describes the technical solutions of the embodiments of the present application through some embodiments. Some of the following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeatedly described in some embodiments.

[0039] The present application provides a target network model, which is configured to perform an identity anonymization process on targets (e.g., faces) in a target image to generate an identity-anonymized image of the target image. Thus, in some embodiments, the target network model may be referred to as an identity anonymizer model or an identity anonymizer. First, we introduce the training process of the target network model.

[0040] 3 is an exemplary flowchart of a model training method according to an embodiment of the present application. An execution subject of the embodiment of the present application is an apparatus having a model training function, such as a model training apparatus, which may be a computing device or may be a part of a computing device. In the following, an execution subject is described as a computing device as an example. As shown in FIG. 3, the method of the embodiment of the present application includes the following steps:

[0041] In step S301, the computing device projects a first training image into a target space by a projection module in a target network model to obtain N (N is a positive integer) first virtual identity vectors.

[0042] The first training image in the embodiment of the present application is a training image in the training data. If the first training image is a face image, the first training image is obtained with the permission of the user, and the collection, use and processing of related image data should comply with the relevant laws, regulations and standards of the relevant countries or regions.

[0043] In the embodiment of the present application, the process of training a model using each first training image is basically the same, and for convenience of explanation, one first training image is taken as an example.

[0044] In the embodiment of the present application, the target network model is used to project the first training image into the target space to obtain one or more virtual identity vectors of the first training image, thereby enabling the target network model to learn the identity information of the first training image. After the target network model has fully learned the identity information, when actually performing the identity anonymization process, it can directly sample the target space of the target network model to generate the virtual identity vectors.

[0045] The present embodiment is primarily concerned with the concepts of attribute vectors and virtual identity vectors.

[0046] Here, the virtual identity vector is a vector corresponding to virtual identity information, and the virtual identity information is identity information after concealing identifiable identity features, for example, face information after concealing identifiable identity features of a face.

[0047] An attribute vector is a vector corresponding to attribute information, and other feature information in an image other than the identifiable identity features, such as background information, is called attribute information. The target network model of the present embodiment can generate independent virtual identity vectors.

[0048] FIG. 4 is an exemplary structural diagram of a target network model according to an embodiment of the present application. As shown in FIG. 4, the target network model of the embodiment of the present application includes a projection module, an attribute module and a fusion module.

[0049] Wherein, the projection module is configured to project the first training image into the target space to obtain N first virtual identity vectors of the first training image, where N is a positive integer, and the embodiment of the present application is not limited to the value of N, which can be set according to actual needs.

[0050] The attribute module is configured to perform attribute vector extraction on the second training image, and extract M attribute vectors of the second training image, where M is a positive integer, and the embodiment of the present application is not limited to the value of M, which can be set according to practical needs. In some embodiments, M is equal to N.

[0051] The fusion module is configured to perform image generation based on the N first virtual identity vectors and the M attribute vectors to obtain an identity-anonymized image of the second training image.

[0052] When N is a positive integer greater than 1, the N first virtual identity vectors each correspond to a different resolution.

[0053] As can be seen from the above, in the target network model of the embodiment of the present application, the projection module is configured to generate a virtual identity vector of the target in the second training image, which virtual identity vector hides the real identity characteristics of the target in the second training image, and the attribute module is configured to extract an attribute vector of the second training image, which attribute vector reserves other features than the real identity characteristics of the target in the second training image. In this way, after the fusion module performs image generation based on the above virtual identity vector and attribute vector, an anonymous image that hides the identity of the target in the second training image, i.e., an identity anonymous image, can be obtained.

[0054] In some embodiments, as shown in FIG. 5, the projection module includes a first projection unit and a second projection unit, and the target space includes a first space Z and a second space W, in this case, the above-mentioned computing device may use the projection module in the target network model to project the first training image into the target space to obtain N first virtual identity vectors in the following manner:

[0055] A priori identity information of a first training image is extracted, and a first projection unit projects the a priori identity information into a first space Z to obtain N identity latent vectors, and a second projection unit projects the N identity latent vectors into a second space W to obtain N first virtual identity vectors.

[0056] As shown in Figure 5, firstly, extract the a priori identity information of the first training image, for example, extract the a priori identity information of the first training image by a pre-trained discriminative model, then project the a priori identity information of the first training image into a first space Z by a first projection unit to obtain N identity latent vectors, and then project the N identity latent vectors into a second space W by a second projection unit to obtain N first virtual identity vectors.

[0057] The above first space Z and second space W may be different latent spaces. The embodiment of the present application is not limited to the first space Z and the second space W. In some embodiments, the first space is a latent space Z, which satisfies a standard Gaussian distribution.

[0058] In this case, the above-mentioned first projection unit can project the a priori identity information into the first space Z to obtain N identity latent vectors in the following manner.

[0059] A first projection unit projects the a priori identity information as the mean and variance of the first space, and samples according to the mean and variance of the first space to obtain N identity latent vectors.

[0060] In some embodiments, the first projection unit is a variational autoencoder (VAE), such as a conditional variational autoencoder (CVAE), which is a generative network that uses an encoder to learn the distribution of data to obtain hidden variables, and then uses a decoder to restore the hidden variables to the original form of the data. The conditional variational autoencoder learns the distribution of data, then samples to generate new data, and is typically used for image generation.

[0061] In this way, the a priori identity information of the first training image is input to the VAE, which can project the a priori identity information as the mean and variance of the first space, and then perform sampling based on the mean and variance of the first space to obtain N identity latent vectors of the first training image.

[0062] In this example, the first space is a latent space Z that satisfies a standard Gaussian distribution. Therefore, in order to enhance the representation capability of the latent space, the embodiment of the present application generates different latent vectors at different resolution levels, for example, N identity latent vectors, which include multiple identity latent vectors.

[0063]

number

[0064] In some embodiments, the second space W is obtained through the potential space Z, for example by performing a linear or non-linear mapping through the potential space Z.

[0065] The embodiment of the present application is not limited to the network structure of the second projection unit, and may be, for example, a mapping network configured by multiple fully connected layers.

[0066] The embodiment of the present application projects the a priori identity information of the first training image into the shadow space (i.e., the target space) of the projection module, so that the projection module can fully learn the identity information of the first training image and subsequently generate a virtual identity vector that matches the actual situation.

[0067] In step S302, an attribute module in the target network model performs attribute vector extraction on the second training image to obtain M attribute vectors (M is a positive integer).

[0068] Here, the second training image is any image in the training data set, and the second training image may be the same image as the first training image or may be a different image. The attribute module of the embodiment of the present application is configured to learn the attribute information of the second training images to generate M attribute vectors. The embodiments of the present application are not limited to a network model of the attribute module.

[0069] In some embodiments, as shown in FIG. 6 , the attribute module includes an encoding unit and a decoding unit, in which case the attribute module in the target network model can perform attribute vector extraction on the second training image to obtain M attribute vectors in the following manner:

[0070] The second training image is input to the encoding unit to obtain the feature information of the second training image, and the feature information is input to the decoding unit to obtain M attribute vectors.

[0071] In some embodiments, the encoding unit includes multiple feature extraction layers and the decoding unit also includes multiple feature extraction units, with a skip connection between at least one feature extraction layer of the encoding unit and at least one feature extraction layer of the decoding unit. After generating N first virtual identity vectors and M attribute vectors based on the above steps, perform the following step S303.

[0072] In step S303, image generation is performed by a fusion module in the target network model according to the N first virtual identity vectors and the M attribute vectors to obtain an identity-anonymized image of the second training image.

[0073] In Example 1, N first virtual identity vectors are spliced ​​to obtain a spliced ​​first virtual identity vector, M attribute vectors are stitched together, and image generation is performed on the spliced ​​first virtual identity vector and the spliced ​​attribute vector, and then input into a fusion module to generate an identity anonymized image.

[0074] For example, the spliced ​​first virtual identity vector is cascaded with the spliced ​​attribute vector and then input into a fusion module to generate an identity anonymized image.

[0075] Further, for example, the spliced ​​first virtual identity vector is added with the spliced ​​attribute vector and then input into a fusion module to generate an identity anonymized image.

[0076] In Example 2, the fusion module includes multiple different resolution layers, in which case the fusion module performs image generation based on the N first virtual identity vectors and M attribute vectors in the following manner to obtain an identity-anonymized image of the second training image.

[0077] Based on the resolutions corresponding to the N first virtual identity vectors, the N first virtual identity vectors are used as patterns and the M attribute vectors are used as noise to input into the corresponding resolution layer to obtain an identity anonymized image of the second training image.

[0078] For example, N is 3, M is 4, and the fusion module includes four different resolution layers, three of which are represented as first virtual identity vector 1, first virtual identity vector 2, and first virtual identity vector 3, and four attribute vectors are represented as attribute vector 1, attribute vector 2, attribute vector 3, and attribute vector 4. The four resolution layers are represented as resolution layer 1, resolution layer 2, resolution layer 3, and resolution layer 4 in sequence according to the size of the resolution. The first virtual identity vector 1 corresponds to the resolution layer 1 and resolution layer 2 with low resolution, the first virtual identity vector 2 corresponds to the resolution layer 3 with medium resolution, and the first virtual identity vector 3 corresponds to the resolution layer 4 with the highest resolution. The four attribute vectors correspond to the four resolution layers in sequence according to the resolution size.

[0079] Exemplarily, the first virtual identity vector 1 is input into the resolution layer 1 to obtain feature information 1, the attribute vector 1 is combined with the feature information 1, and then input into the resolution layer 2 together with the first virtual identity vector 2 to obtain feature information 2. The attribute vector 2 is combined with the feature information 2, and then input into the resolution layer 3 together with the first virtual identity vector 3 to obtain feature information 3. The attribute vector 3 is combined with the feature information 3, and then input into the resolution layer 4 together with the first virtual identity vector 4 to obtain feature information 4. Finally, after processing such as combining the feature information 4 with the attribute vector 4, an identity anonymized image of the second training image is generated.

[0080] In some embodiments, the fusion module is a style-based generator (StyleGAN2). As shown in FIG. 7, an AdaIN layer is included between two resolution layers adjacent to the fusion module, for example, an affine transform (AT) is performed on the first virtual identity vector i+1, the feature information i output by the i-th resolution layer is synthesized with the attribute vector i, and the affine transformed first virtual identity vector i+1 is input to the AdaIN layer, an AdaIN operation is performed, and the AdaIN operation result is input to the i+1-th resolution layer.

[0081] The fusion module of the embodiments of the present application may be an adversarial model such as StyleGAN3 and ProGAN, and when the fusion module adopts a different adversarial model, the manner of determining the identity anonymized image of the second training image may be different, and the embodiments of the present application are not limited thereto.

[0082] In some embodiments, the first projection unit is a VAE, the second projection unit is a mapping network, the attribute module is an autoencoder, and the fusion module is StyleGAN2 as an example to introduce the model training process of the embodiments of the present application.

[0083] Exemplarily, as shown in FIG. 8, a pre-trained face identification model is used to generate a priori identity information based on a first training image Xs. Then, the priori identity information is input to a VAE, and the priori identity information is projected into a first space Z by the VAE to obtain N identity latent vectors, for example, three identity latent vectors are obtained, and the three identity latent vectors correspond to three different resolutions: low, medium, and high. Then, the N identity latent vectors are input to a mapping network, and the N identity latent vectors are projected from the first space Z to a second space W through the mapping network to obtain N first virtual identity vectors. Furthermore, the second training image Xt is input to an autoencoder, and M attribute vectors are generated after processing the second training image Xt by the autoencoder. Finally, the M attribute vectors are used as noise and the N first virtual identity vectors are used as patterns to input into each layer of StyleGAN2, and an identity anonymized image Ys,t of the second training image output by StyleGAN2 is obtained.

[0084] Based on the above steps, the first training image and the second training image are input into the target network model, and an identity anonymized image of the second training image output by the target network model can be obtained. Then, the target network model is trained by performing the following step S304.

[0085] In step S304, a loss for a target network model is determined based on the identity anonymized image, and the target network model is trained based on the loss.

[0086] Based on the above steps, the target network model outputs an identity-anonymized image of the second training image, and determines a loss for the target network model based on the identity-anonymized image.

[0087] In some embodiments, the identity anonymized image is input to a judgment model, which is a pre-trained model that can predict the anonymization degree of the identity anonymized image. For example, the identity anonymized image is input to the judgment model, which performs identity identification on the identity anonymized image, and determines the identification result as the loss of the target network model. If the identification accuracy is high, it means that the anonymization effect of the current target network model is not ideal, and in this case, the parameters in the target network model are adjusted according to the loss of the target network model. Then, new first and second training images are selected to perform the above steps S301 to S304, and the target network model is continuously trained until the target network model meets the training end condition. Here, the training end condition includes at least the number of training times reaching a predetermined number of times or the anonymization degree of the model reaching an expected effect.

[0088] In some embodiments, when the first space Z shown in FIG. 5 is a latent space that satisfies a standard Gaussian distribution, the embodiments of the present application apply a KL divergence constraint to the N identity latent vectors in the first space Z.

[0089]

number

[0090] Based on this, the embodiment of the present application may further determine a divergence constraint for the N identity latent vectors, in which case the step of determining a loss for the target network model based on the identity anonymized image includes the step of determining a loss for the target network model based on the identity anonymized image and the divergence constraint. For example, the divergence constraint of N identity latent vectors is expressed as follows:

[0091]

number

[0092]

number

[0093] where μi is the mean corresponding to the i-th identity latent vector among the N identity latent vectors, and σi is the variance corresponding to the i-th identity latent vector among the N identity latent vectors.

[0094] It should be noted that the above formula (1) is merely one example, and the method for determining the divergence constraints of the N identity latent vectors in the embodiments of the present application is not limited to the above formula (1), and may be, for example, a method for calculating other divergence constraints, such as by modifying the above formula (1). In the present embodiment, the divergence constraints for N identity latent vectors are

[0095]

number

[0096] In this way, the training not only enables the projection module to fully learn the identity information, but also makes the first space of the projection module satisfy a standard Gaussian distribution, so that later, during the anonymization process, the first space can be directly sampled to generate N identity latent vectors that satisfy the standard Gaussian distribution to generate a virtual identity vector.

[0097] In some embodiments, the second space is obtained by nonlinear mapping of the first space, which is a complex non-Gaussian distribution. As shown in FIG. 5, after mapping the identity information into the first space, the second space of the intermediate latent space in this case is

[0098]

number

[0099] We found that the distribution of is not uniform, and the real identity vectors are centered in different centers and do not overlap with the generated virtual identity vector, so the virtual identity vector cannot generate a reasonable face identity. Therefore, the embodiment of the present application uses a contrast loss to estimate the second space, i.e.

[0100]

number

[0101] We propose to constrain the latent vectors in the space (i.e., the first virtual identity vectors) such that latent vectors from the same identity are aggregated together and latent vectors of different identities repel each other, such that all latent vectors are evenly distributed throughout the space.

[0102] On this basis, the embodiment of the present application can further determine the identity loss by the following steps. In step 1, a third training image is acquired. In step 2, the third training image is processed by the projective reference module to obtain N second virtual identity vectors. In step 3, an identity loss is determined based on the N first virtual identity vectors and the N second virtual identity vectors.

[0103] The above third training image and the first training image are both two different images of the first target, for example, the third training image and the first training image are two different face images of the same user.

[0104] The above-mentioned projection reference module has the same network structure as the projection module, and is updated according to the projection module. For example, the projection reference module is updated according to the momentum of the projection module, that is, the projection reference module is updated slowly along with the update of the projection module. Exemplarily, the projective reference module may be updated according to the following equation (2): Pθ'(t)=(1-Δ)*Pθ'(t-1)+Δ*Pθ(t) (2)

[0105] where Pθ'(t) is the projection reference module parameter after the tth update, Pθ'(t-1) is the projection reference module parameter after the t-1th update, Pθ(t) is the projection module parameter after the tth update, and Δ is a small value such as 0.01.

[0106] As shown in Fig. 9, in the model training process, in order to determine the identity loss, the embodiment of the present application constrains the first virtual identity vector output by the projection module by arranging one projection reference module that is completely consistent with the network structure of the projection module. Illustratively, the first training image is input to the projection module to obtain N first virtual identity vectors of the first training image, and the third training image is input to the projection reference module to obtain N second virtual identity vectors of the third training image. Because the first training image and the third training image are images of the same target, and the network structures of the projection module and the projection reference module are consistent, after the model training is completed, the difference between the N first virtual identity vectors corresponding to the first training image and the N second virtual identity vectors is relatively small. Under this condition, the projection module in the target network model can be trained based on the N first virtual identity vectors and the N second virtual identity vectors corresponding to the first training image, so that the projection module can generate a virtual identity vector that meets the requirements.

[0107] In the above step 3, the manner of determining the identity loss based on the N first virtual identity vectors and the N second virtual identity vectors includes but is not limited to the following:

[0108] In method 1, the differences between N first virtual identity vectors and N second virtual identity vectors at different resolutions are determined, and the sum of the differences or the average of the differences is determined as the identity loss. For example, when N is 3, the difference 1 between the first virtual identity vector 1 and the second virtual identity vector 1 is determined, the difference 2 between the first virtual identity vector 2 and the second virtual identity vector 2 is determined, and the difference 3 between the first virtual identity vector 3 and the second virtual identity vector 3 is determined. The sum of the difference 1, the difference 2, and the difference 3 is determined as the identity loss, or the average of the difference 1, the difference 2, and the difference 3 is determined as the identity loss.

[0109] In method 2, the embodiment of the present application has N dynamic lists

[0110]

number

[0111]

number

[0112] In this case, based on the N first virtual identity vectors and the N second virtual identity vectors, the identity loss can be determined by the following steps:

[0113] In step 31, for an i-th first virtual identity vector among the N first virtual identity vectors, the i-th second virtual identity vector is used to update the virtual identity vector corresponding to the first target in the i-th dynamic list. Here, the i-th dynamic list contains the virtual identity vectors of different targets at the i-th resolution, where i is a positive integer from 1 to N.

[0114] In an embodiment of the present application, in the N second virtual identity vectors, each of the N second virtual identity vectors corresponds to one dynamic list, for example, when N is 3, they correspond to low resolution, medium resolution and high resolution, respectively, and thus the dynamic list includes three: a first dynamic list corresponding to low resolution, a second dynamic list corresponding to medium resolution, and a third dynamic list corresponding to high resolution.

[0115] Assuming i=1, the first second virtual identity vector is used to update the virtual identity vector corresponding to the first target in the first dynamic list.

[0116] Assuming that i=2, the second virtual identity vector is used to update the virtual identity vector corresponding to the first target in the second dynamic list.

[0117] Assuming that i=3, the third second virtual identity vector is used to update the virtual identity vector corresponding to the first target in the third dynamic list.

[0118] In step 32, an identity sub-loss corresponding to the i-th first virtual identity vector is determined based on the i-th first virtual identity vector and the updated i-th dynamic list.

[0119] Exemplarily, as shown in FIG. 10, the first training image and the third training image are two different images of the first target j, the first training image Xj is input to the projection module to obtain N first virtual identity vectors Wj, and the third training image Xj' is input to the projection reference module to obtain N second virtual identity vectors Wj'. For the i-th resolution of the N resolutions, the i-th dynamic list Ki includes second virtual identity vectors of different targets at the i-th resolution, and the i-th dynamic list Ki is updated in real time. Exemplarily, the i-th second virtual identity vector is used to update the virtual identity vector kj corresponding to the first target j in the i-th dynamic list Ki, i.e., kj is updated to Wj'. Then, the identity sub-loss i corresponding to the i-th first virtual identity vector is determined based on the i-th second virtual identity vector and the updated i-th dynamic list. In the embodiment of the present application, there is no limitation on the manner of determining the identity sub-loss corresponding to the i-th first virtual identity vector.

[0120] For example, using a loss scheme such as center loss or triplet loss, an identity sub-loss corresponding to the i-th first virtual identity vector is determined based on the i-th first virtual identity vector and the updated i-th dynamic list.

[0121] In some embodiments, the above step of determining an identity sub-loss corresponding to the i-th first virtual identity vector among the N first virtual identity vectors may include the following steps.

[0122] In step 321, a first ratio between the i-th second virtual identity vector and a first predetermined value is obtained, the first ratio is multiplied by the i-th first virtual identity vector to obtain a first result, and an exponential operation is performed on the first result to obtain a first operation value.

[0123] In step 322, a second ratio between each second virtual identity vector and a first predetermined value is obtained in the updated i-th dynamic list, and for each of the second ratios, the second ratio is multiplied by the corresponding i-th first virtual identity vector to obtain a second result, and an exponentiation operation is performed on the second result to obtain a second operation value corresponding to each second virtual identity vector.

[0124] In step 323, a sum of the second operations corresponding to each second virtual identity vector is determined, a third ratio of the first operation value to the sum is obtained, and a logarithmic operation is performed on the third ratio to obtain a third operation value.

[0125] In step 324, the negative of the third calculation value is determined as the identity sub-loss corresponding to the i-th first virtual identity vector. For example,

[0126]

number

[0127]

number

[0128]

number

[0129] are positive samples, the rest are negative samples, and we use a contrast loss in the form of Information Noise Contrastive Noise (InfoNCE) to estimate the identity sub-loss.

[0130]

number

[0131]

number

[0132]

number

[0133]

number

[0134]

number

[0135]

number

[0136] In step 33, the sum of the identity sub-losses corresponding to each of the N first virtual identity vectors is determined as the identity loss of the target network model.

[0137] After determining the identity sub-loss corresponding to the i-th first virtual identity vector according to step 32 above, the sum of the identity sub-losses corresponding to each of the N first virtual identity vectors is determined as the identity loss. For example, if N is 3, the identity sub-loss corresponding to each first virtual identity vector among the three first virtual identity vectors is determined according to the above method, and the sum of the identity sub-losses corresponding to the three first virtual identity vectors is determined as the identity loss of the model.

[0138] In an embodiment of the present application, after determining the identity loss in the model training process according to the above method, the step of determining the loss of the target network model based on the identity anonymized image and the divergence constraint includes the following steps: Based on the identity anonymized image, the divergence constraint and the identity loss, we determine the loss of the target network model.

[0139] Exemplarily, a reconstruction loss between the identity anonymized image and a second training image is determined, and a loss for a target network model is determined based on the reconstruction loss, the divergence constraint and the identity loss.

[0140] In one example, the difference between the identity-anonymized image and the second training image is determined as the reconstruction loss, e.g., the sum of the differences between each pixel point of the identity-anonymized image and the corresponding pixel point of the second training image is determined as the reconstruction loss. In another example, the reconstruction loss L rec Determine.

[0141]

number

[0142]

number

[0143] Based on the above steps, the reconstruction loss L rec After determining, determine a loss for the target network model based on the reconstruction loss, the divergence constraint, and the identity loss. For example, determine a weighted sum of the reconstruction loss, the divergence constraint, and the identity loss as the final loss for the target network model.

[0144] In some embodiments, to improve the training accuracy of the model, embodiments of the present application further include a step of determining an identity contrast loss of the identity anonymized image, illustratively comprising the steps of:

[0145] In step A, a first distance between the identity-anonymized image and a first training image, a second distance between the identity-anonymized image and a second training image, and a third distance between the first training image and the second training image are determined. In step B, a contrast loss is determined based on the first distance, the second distance, and the third distance. Here, the first distance, the second distance, and the third distance may be determined by any distance method, such as the cosine distance. In Example 1, after determining the first distance, the second distance, and the third distance according to step A, the sum of the first distance, the second distance, and the third distance is determined as the contrast loss. In example 2, the square of the difference between the second distance and the third distance is summed with the first distance and the difference between a predetermined value and the sum is determined as the contrast loss. In one example, the contrast loss L is calculated based on the following equation (5): ICL Determine.

[0146]

number

[0147]

number

[0148]

number

[0149]

number

[0150] Based on the above steps, the contrast loss L ICL After determining, determine a loss for the target network model based on the reconstruction loss, the divergence constraint, the identity loss, and the contrast loss, for example, determine a weighted sum of the reconstruction loss, the divergence constraint, the identity loss, and the contrast loss as the loss for the target network model.

[0151] In some embodiments, when the fusion module is an adversarial network, the model training process further determines an adversarial loss for the model, for example, determining the adversarial loss based on the identity anonymized images and the first training images. For example, the adversarial loss L GAN Determine.

[0152]

number

[0153] where D is the discriminator, G is the generator, E(*) represents the expectation of the distribution function, and D(X S ) is the first training image X S The classification result is D(Y s,t ) is the identity anonymized image Y s,t This is the identification result.

[0154] Based on the above steps, the adversarial loss L GAN After determining, a loss for the target network model can be determined based on the reconstruction loss, the divergence constraint, the identity loss, the contrast loss, and the adversarial loss, for example, a weighted sum of the reconstruction loss, the divergence constraint, the identity loss, the contrast loss, and the adversarial loss is determined as the loss for the target network model.

[0155] It should be noted that the embodiments of the present application do not limit the sizes of the weight values ​​corresponding to the reconstruction loss, the divergence constraint, the identity loss, the contrast loss, and the adversarial loss, and the sizes can be determined according to actual needs.

[0156] In some embodiments, a weighted operation is performed on the reconstruction loss, the divergence constraint, the identity loss, the contrast loss, and the adversarial loss based on the following formula (7) to obtain the loss L of the target network model. total can be obtained.

[0157]

number

[0158] The weights corresponding to each loss in the above formula (7) are just an example, and the weights corresponding to each loss in the embodiments of the present application include, but are not limited to, those shown in the above formula (7) and can be determined as needed.

[0159] In some embodiments, in order to improve the training accuracy of the target network model, other losses besides the losses described in the above embodiments can be determined, and the embodiments of the present application are not limited thereto and can be determined according to actual needs.

[0160] As can be seen from the above description, the embodiment of the present application can realize identity anonymization by generating a first virtual identity vector corresponding to different resolutions, and can improve the resolution of anonymization, for example, 1024 2 It produces high resolution anonymized results with fewer image artifacts and higher fidelity.Furthermore, in the embodiment of the present application, the model training does not rely on key regression and segmentation models, does not remove face regions in the image, and preserves pose, details and occlusions in the original photo.

[0161] In the application process of the target network model, the embodiment of the present application projects the first training image into the target space by the projection module to obtain N first virtual identity vectors, so that the target network model can fully learn the identity information in the image, performs attribute vector extraction on the second training image to obtain M attribute vectors, so that the target network model can fully learn the attribute information in the image, and performs image generation according to the N first virtual identity vectors and the M attribute vectors to obtain an identity anonymized image of the second training image, so that the model obtained by training can generate an image carrying virtual identity information without changing the attribute information of the original image. That is, the present application provides a new target network model, and through the above training method, the target network model learns the identity information in the first training image, so that the target network model can independently generate virtual identities and fully learn the attribute information in the second training image. In the overall learning process, there is no need to remove the face region in the image or use real identity information for guidance, but a clear supervision target in the face changing task can be utilized to train the target network model, improve the fidelity and resolution of the identity anonymization generation of the target network model, and generate high-quality identity anonymized images with the trained target network model.

[0162] The model training method of the present application has been specifically introduced above with reference to FIGS. 3 to 10. Hereinafter, the identity anonymization method of the present application will be specifically described with reference to FIGS. 11 to 13.

[0163] 11 is an exemplary flowchart of an identity anonymization method according to an embodiment of the present application. The identity anonymization method shown in FIG. 11 is a method for performing identity anonymization processing using the above trained target network model. As shown in FIG. 11, the method includes the following steps:

[0164] In step S401, sampling is performed in the target space of the projection module in the target network model to obtain N (N is a positive integer) virtual identity vectors.

[0165] As can be seen from the above embodiment, the embodiment of the present application uses a first training image to train the projection module, so that the projection module can fully learn the identity information in the first training image, and then, in actual use, N virtual identity vectors can be obtained by sampling the target space of the projection module.

[0166] The embodiments of the above step S401 include, but are not limited to, some of the following: In method 1, sampling is performed based on the mean and variance of the target space of the trained projection module to obtain N virtual identity vectors. For example, random sampling is performed on the variance of the target space, and then added to the mean of the target space to obtain one virtual identity vector, and the above steps can be repeated to obtain N virtual identity vectors.

[0167] In method 2, the target space includes a first space and a second space, and the target network model includes a second projection unit. In this case, sampling can be performed in the target space of the projection module in the target network model to obtain N virtual identity vectors in the following manner:

[0168] Sampling is performed in the first space to obtain N identity latent vectors, and the N identity latent vectors are projected into the second space by a second projection unit to obtain N virtual identity vectors.

[0169] In the embodiment of the present application, when actually performing anonymization, the first projection unit of the projection module is not used, and only the second projection unit of the projection module is used for projection. For example, as shown in FIG. 12, sample in the first space Z that meets the standard Gaussian distribution to obtain N identity latent vectors, and then input the N identity latent vectors into the second projection unit. The second projection unit projects the N identity latent vectors into the W space to obtain N virtual identity vectors. FIG. 12 shows an example in which N is 3 and the second projection unit is a mapping network, but the projection module of the embodiment of the present application is not limited by what is shown in FIG. 12.

[0170] As can be seen from the above description, the first space is trained using the first training image, so that the variance and mean of the first space meet the standard Gaussian distribution. In this way, first, sampling is performed in the first space to generate N identity latent vectors, for example, sampling is performed according to the mean and variance of the first space to obtain N identity latent vectors, random sampling is performed on the variance of the first space, and then added to the mean of the first space to obtain one identity latent vector, and the above steps are repeated to obtain N identity latent vectors. Then, the second projection unit projects the N identity latent vectors into the second space to obtain N virtual identity vectors.

[0171] In some embodiments, the N virtual identity vectors each correspond to a different resolution, e.g., N=3, where the first virtual identity vector corresponds to a low resolution, the second virtual identity vector corresponds to a medium resolution, and the third virtual identity vector corresponds to a high resolution.

[0172] After obtaining the N virtual identity vectors based on the above method, the following steps S402 and S403 are carried out to obtain an identity anonymized image of the target image.

[0173] In step S402, an attribute vector is extracted for the image to be processed by an attribute module in the target network model, and M attribute vectors (M is a positive integer) are obtained. The attribute module of the present embodiment is configured to extract attribute information in a target image.

[0174] In some embodiments, the attribute module includes an encoding unit and a decoding unit, in which case attribute vector extraction can be performed on the target image in the following manner to obtain M attribute vectors. The image to be processed is input to the encoding unit, feature information of the image to be processed is obtained, and the feature information is input to the decoding unit to obtain M attribute vectors.

[0175] In some embodiments, the encoding unit may include multiple feature extraction layers, and similarly, the decoding unit may also include multiple feature extraction layers, where the feature extraction layers may include convolutional layers, etc.

[0176] In some embodiments, there is a skip connection between at least one feature extraction layer of the encoding unit and at least one feature extraction layer of the decoding unit. The above generated M attribute vectors can correspond to different resolutions. In some embodiments, the target network model is an autoencoder.

[0177] In step S403, an image is generated by a fusion module in the target network model according to the N virtual identity vectors and the M attribute vectors to obtain an identity-anonymized image of the target image.

[0178] Based on the above steps, N virtual identity vectors and M attribute vectors are generated and input into a fusion module to obtain an identity anonymized image of the target image. The above S403 implementations include, but are not limited to, some of the following:

[0179] In example 1, N virtual identity vectors are spliced ​​together, and M attribute vectors are spliced ​​together at the same time. The spliced ​​virtual identity vectors and attribute vectors are fused and then input into the fusion module. For example, the spliced ​​virtual identity vector is cascaded with the attribute vector and then input into the fusion module. Further, for example, the spliced ​​virtual identity vector is added with the attribute vector before being input to the fusion module.

[0180] In example 2, the fusion module includes multiple different resolution layers, in which, based on the resolution corresponding to the N virtual identity vectors, the N virtual identity vectors are used as patterns and the M attribute vectors are used as noises to input into the corresponding resolution layers, so as to obtain an identity anonymized image of the image to be processed.

[0181] In some embodiments, the fusion module is StyleGAN2. In this case, as shown in FIG. 7, an AdaIN layer is included between two adjacent resolution layers of the fusion module, for example, an affine transformation is performed on the virtual identity vector i+1, the feature information i output by the i-th resolution layer is combined with the attribute vector i, and the affine transformed virtual identity vector i+1 is input to the AdaIN layer, an AdaIN operation is performed, and the AdaIN operation result is input to the i+1-th resolution layer.

[0182] The fusion module of the embodiment of the present application may be an adversarial model such as StyleGAN3 and ProGAN. In some embodiments, the second projection unit is a mapping network, the attribute module is an autoencoder, and the fusion module is StyleGAN2 as an example to introduce the identity anonymization process of the embodiment of the present application.

[0183] Exemplarily, as shown in FIG. 13, the projection module samples in the first space Z to obtain N identity latent vectors, for example, obtain three N identity latent vectors, which correspond to three different resolutions, namely, low, medium, and high. Then, the N identity latent vectors are input into a mapping network, and the N identity latent vectors are projected from the first space Z to the second space W through the mapping network to obtain N virtual identity vectors. Furthermore, the target image Xt is input into an autoencoder, and the target image Xt is processed through the autoencoder to generate M attribute vectors. Finally, the M attribute vectors are used as noise and the N virtual identity vectors are used as patterns to input into each layer of StyleGAN2, and an identity anonymized image Ys,t of the target image output by StyleGAN2 is obtained.

[0184] In the identity anonymization method according to the embodiment of the present application, a projection module in a target network model performs sampling in a target space to obtain N virtual identity vectors, an attribute module in the target network model performs attribute vector extraction on the image to be processed to obtain M attribute vectors, and a fusion module in the target network model performs image generation based on the N virtual identity vectors and the M attribute vectors to obtain an identity anonymized image of the image to be processed. That is, the target network model of the embodiment of the present application can independently generate a virtual identity, and when performing identity anonymization on the image to be processed, the fidelity of identity anonymization is improved without the need to remove face regions in the image to be processed.

[0185] Above, the method embodiment of the present application is described in detail with reference to Fig. 3 to Fig. 13, and below, the apparatus embodiment of the present application is described with reference to Fig. 14 to Fig. 15. Fig. 14 is an exemplary block diagram of a model training apparatus according to an embodiment of the present application. The training apparatus 10 may be a computing device or may be a part of a computing device. As shown in Fig. 14, the model training apparatus 10 includes a projection unit 11, an attribute unit 12, a fusion unit 13, and a training unit 14,

[0186] The projection unit 11 is configured to project a first training image into a target space by a projection module in a target network model to obtain N first virtual identity vectors, where N is a positive integer;

[0187] The attribute unit 12 is configured to perform attribute vector extraction on the second training image through an attribute module in the target network model to obtain M attribute vectors, where M is a positive integer;

[0188] The fusion unit 13 is configured to perform image generation according to the N virtual identity vectors and the M attribute vectors through a fusion module in the target network model to obtain an identity anonymized image of the second training image;

[0189] The training unit 14 is configured to determine a loss for the target network model based on the identity-anonymized image, and to train the target network model based on the loss.

[0190] In some embodiments, the projection module includes a first projection unit and a second projection unit, the target space includes a first space and a second space, and the projection unit 11 is further configured to extract a priori identity information of the first training image, project the a priori identity information into the first space by the first projection unit to obtain N identity latent vectors, and project the N identity latent vectors into the second space by the second projection unit to obtain the N first virtual identity vectors.

[0191] In some embodiments, the projection unit 11 is further configured to project the a priori identity information as a mean and a variance of the first space by the first projection unit, and perform sampling based on the mean and variance of the first space to obtain the N identity latent vectors.

[0192] In some embodiments, training unit 14 is further configured to determine a divergence constraint for the N identity latent vectors, and determine a loss for the target network model based on the identity anonymized image and the divergence constraint. In some embodiments, each of the N first virtual identity vectors corresponds to a different resolution. In some embodiments, the first projection unit is a variational autoencoder.

[0193] In some embodiments, the training unit 14 is further configured to acquire a third training image, where the third training image and the first training image are both two different images of a first target, project the third training image into a target space by a projection reference module of the target network model, and obtain N second virtual identity vectors, the projection reference module having the same network structure as the projection module and being updated according to the projection module, determine an identity loss based on the N first virtual identity vectors and the N second virtual identity vectors, and determine a loss of the target network model based on the identity anonymized image, the divergence constraint and the identity loss.

[0194] In some embodiments, the training unit 14 is further configured to: for an i-th second virtual identity vector (i is a positive integer from 1 to N) among the N second virtual identity vectors, use the i-th second virtual identity vector to update a virtual identity vector corresponding to the first target in an i-th dynamic list, where the i-th dynamic list includes virtual identity vectors of different targets at the i-th resolution; determine an identity sub-loss corresponding to the i-th first virtual identity vector based on the i-th first virtual identity vector and the updated i-th dynamic list; and determine the sum of the identity sub-losses corresponding to each of the N first virtual identity vectors as the identity loss.

[0195] In some embodiments, training unit 14 is further configured to: obtain a first ratio between the i-th second virtual identity vector and a first predetermined value; multiply the first ratio by the i-th first virtual identity vector to obtain a first result; perform an exponentiation operation on the first result to obtain a first operation value; obtain a second ratio between each second virtual identity vector in the updated i-th dynamic list and a first predetermined value; for each said second ratio, multiply the second ratio by a corresponding i-th first virtual identity vector to obtain a second result; perform an exponentiation operation on the second result to obtain a second operation value corresponding to each of the second virtual identity vectors; determine a sum of the second operation values ​​corresponding to each of the second virtual identity vectors; obtain a third ratio between the first operation value and the sum; perform a logarithmic operation on the third ratio to obtain a third operation value; and determine the negative of the third operation value as the identity sub-loss corresponding to the i-th first virtual identity vector.

[0196] In some embodiments, the attribute module includes an encoding unit and a decoding unit, and the attribute unit 12 is further configured to perform feature extraction on the second training image by the encoding unit to obtain feature information of the second training image, and decode the feature information by the decoding unit to obtain M attribute vectors.

[0197] In some embodiments, there is a skip connection between at least one feature extraction layer of the encoding unit and at least one feature extraction layer of the decoding unit.

[0198] In some embodiments, the fusion module includes multiple different resolution layers, and the fusion unit 13 is further configured to, based on a resolution corresponding to the N first virtual identity vectors, use the N first virtual identity vectors as patterns and the M attribute vectors as noise to input into a corresponding resolution layer to obtain an identity anonymized image of the second training image.

[0199] In some embodiments, training unit 14 is further configured to determine a reconstruction loss between the identity anonymized image and the second training image, and determine a loss for the target network model based on the reconstruction loss, the divergence constraint and the identity loss.

[0200] In some embodiments, training unit 14 is further configured to determine a first distance between the identity anonymized image and the first training image, a second distance between the identity anonymized image and the second training image, and a third distance between the first training image and the second training image, determine a contrast loss based on the first distance, the second distance, and the third distance, and determine a loss for the target network model based on the reconstruction loss, the divergence constraint, the identity loss, and the contrast loss.

[0201] In some embodiments, the training unit 14 is further configured to determine the sum of the square of the difference between the second distance and the third distance and the first distance, and to determine the difference between a predetermined value and the sum as the contrast loss.

[0202] In some embodiments, when the fusion module is an adversarial network, training unit 14 is further configured to determine an adversarial loss based on the identity anonymized images and the first training images, and determine a weighted sum of the reconstruction loss, the divergence constraint, the identity loss, the contrast loss and the adversarial loss as a loss for the target network model.

[0203] It should be understood that the apparatus embodiment and the method embodiment can correspond to each other, and similar descriptions can refer to the method embodiment. In order to avoid repetition, details will not be described again in this specification. Illustratively, the apparatus shown in Fig. 14 can execute the embodiment of the model training method shown in Fig. 3 above, and the above-mentioned and other operations and / or functions of each individual module in the apparatus are respectively for executing the corresponding method embodiment in a computing device, and will not be described again here for brevity.

[0204] 15 is an exemplary block diagram of an identity anonymization device according to an embodiment of the present application. The identity anonymization device 20 may be a computing device or may be part of a computing device. As shown in FIG. 15, the identity anonymization device 20 includes a sampling unit 21, an attribute unit 22, and an anonymization unit 23;

[0205] The sampling unit 21 is configured to sample in a target space of a projection module in a target network model to obtain N virtual identity vectors, where N is a positive integer;

[0206] The attribution unit 22 is configured to perform attribute vector extraction for the processing target image through an attribution module in a target network model, and obtain M attribute vectors (M is a positive integer);

[0207] The anonymization unit 23 is configured to perform image generation based on the N virtual identity vectors and the M attribute vectors by a fusion module in the target network model, to obtain an identity-anonymized image of the processing target image.

[0208] In some embodiments, the target space includes a first space and a second space, the target network model includes a second projection unit, and the sampling unit 21 is further configured to perform sampling in the first space to obtain N identity latent vectors, and project the N identity latent vectors into a second space by the second projection unit to obtain the N virtual identity vectors.

[0209] In some embodiments, the mean and variance of the first space satisfy a standard Gaussian distribution, and the sampling unit 21 is further configured to perform sampling based on the mean and variance of the first space to obtain the N identity latent vectors. In some embodiments, each of the N virtual identity vectors corresponds to a different resolution.

[0210] In some embodiments, the attribute module comprises an encoding unit and a decoding unit, and the attribute unit 22 is further configured to perform feature extraction on the target image by the encoding unit to obtain feature information of the target image, and decode the feature information by the decoding unit to obtain M attribute vectors.

[0211] In some embodiments, there is a skip connection between at least one feature extraction layer of the encoding unit and at least one feature extraction layer of the decoding unit.

[0212] In some embodiments, the fusion module includes a number of different resolution layers, and the anonymization unit 23 is further configured to, based on a resolution corresponding to the N virtual identity vectors, use the N virtual identity vectors as patterns and the M attribute vectors as noise to input into a corresponding resolution layer, to obtain an identity anonymized image of the image to be processed.

[0213] It should be understood that the device embodiment and the method embodiment can correspond to each other, and similar descriptions can refer to the method embodiment. In order to avoid repetition, details will not be described again in this specification. By way of example, the device shown in Fig. 15 can execute the embodiment of the identity anonymization method shown in Fig. 11 above, and the above and other operations and / or functions of each individual module in the device are respectively for executing the corresponding method embodiment in a computing device, and will not be described again here for the sake of brevity.

[0214] The above describes the apparatus of the embodiment of the present application from the perspective of a functional module with reference to the drawings. It should be understood that the functional module may be realized through the form of hardware, through instructions in the form of software, or through a combination of hardware and software modules. For example, each step of the method embodiment in the embodiment of the present application can be completed through an integrated logic circuit of the hardware of the processor and / or instructions in the form of software, and the steps of the method disclosed in the embodiment of the present application may be directly executed by the hardware decoding processor, or may be executed by forming hardware and software modules in the decoding processor. Exemplarily, the software module can be arranged in a conventional storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is arranged in a memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with the hardware.

[0215] 16 is an exemplary block diagram of a computing device according to an embodiment of the present application, the computing device being configured to execute the above method embodiment. As shown in FIG. 16, the computing device 30 includes:

[0216] The computer system may include a memory 31 and a processor 32, and the memory 31 is configured to store a computer program 33 and transmit the program code 33 to the processor 32. In other words, the processor 32 can realize the method in the embodiment of the present application by calling and executing the computer program 33 from the memory 31. For example, the processor 32 may be configured to carry out the above method steps according to instructions in the computer program 33 . In some embodiments of the present application, the processor 32 may include:

[0217] This may include, but is not limited to, a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like.

[0218] In some embodiments of the present application, the memory 31 includes: This may include, but is not limited to, volatile memory and / or non-volatile memory. Here, non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), used as an external cache. By way of illustrative, but non-limiting example, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synch link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM).

[0219] In some embodiments of the present application, the computer program 33 can be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to complete the method of the present application. The one or more modules may be a series of computer program instruction segments that can complete a specific function, and the instruction segments are used to describe the execution process of the computer program 33 in the computing device.

[0220] As shown in FIG. 16, the computing device 30 further comprises: A transceiver 34 may be included, which may be connected to the processor 32 or the memory 31 .

[0221] Here, the processor 32 can control the transceiver 34 to communicate with other devices, for example, to transmit information or data to other devices or to receive information or data transmitted by other devices. The transceiver 34 can include a transmitter and a receiver. The transceiver 34 can also include an antenna, the number of which can be one or more.

[0222] The components of the computing device 30 are connected by a bus system that may include a power bus, a control bus, and a status signal bus in addition to a data bus.

[0223] An embodiment of the present application provides a computer storage medium having stored thereon a computer program which, when executed by a computer, causes the computer to perform the method of the above method embodiment, or an embodiment of the present application further provides a computer program product comprising instructions, when executed by a computer, causing the computer to perform the method of the above method embodiment.

[0224] An embodiment of the present application provides a computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer readable storage medium, a processor of a computing device reading the computer instructions from the computer readable storage medium, and the processor executing the computer instructions to cause the computing device to perform the method of the above method embodiment.

[0225] The above content is merely a specific embodiment of the present application, and the scope of protection of the present application is not limited thereto. Any modifications or replacements that a person skilled in the art can easily think of within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims. [Explanation of symbols]

[0226] 10 Model training device 11 Projection Unit 12 Attribute Units 13 Fusion Unit 14 Training Units 20 Identity Anonymizer 21 Sampling Unit 22 Attribute Units 23 Anonymization Unit 30 Computing Devices 31 Memory 32 processors 33 Program Code 33 Computer Programs 34 Transceiver 101 User Equipment 102 Data Collection Equipment 103 Training equipment 104 Execution Equipment 105 Database 106 Content Library 107 I / O Interface 108 Target Network Model 109 Computing Module

Claims

1. 1. A method for training a model executed by a computing device, comprising: Extracting a priori identity information of a first training image; Projecting the a priori identity information into a first space of a target space by a first projection unit of a projection module in a target network model to obtain N identity latent vectors, where N is a positive integer; Projecting the N identity latent vectors into a second space of the target space by a second projection unit of a projection module in the target network model to obtain N first virtual identity vectors; Performing attribute vector extraction on the second training image by an attribute module in the target network model to obtain M attribute vectors of the second training image, where M is a positive integer; Performing image generation according to the N first virtual identity vectors and the M attribute vectors by a fusion module in the target network model to obtain an identity-anonymized image of the second training image; determining a loss for the target network model based on the identity-anonymized image; and training the target network model based on the loss.

2. The step of obtaining the N identity latent vectors comprises: projecting the a priori identity information as a mean and a variance in the first space by the first projection unit; and sampling the first space based on a mean and a variance to obtain the N identity latent vectors.

3. The model training method includes: determining a divergence constraint for the N identity latent vectors; Determining a loss for the target network model based on the identity anonymized image includes: The method of claim 1 , further comprising determining a loss for the target network model based on the identity-anonymized images and the divergence constraint.

4. The model training method includes: acquiring a third training image, the third training image and the first training image being two different images of a first target; projecting the third training image into a target space by a projection and reference module of the target network model to obtain N second virtual identity vectors, where the projection and reference module has the same network structure as the projection module and is updated according to the projection module; determining an identity loss based on the N first virtual identity vectors and the N second virtual identity vectors; Determining a loss for the target network model based on the identity anonymized image and the divergence constraint, The method of claim 3 , further comprising determining a loss for the target network model based on the identity anonymized images, the divergence constraint, and the identity loss.

5. Determining an identity loss based on the N first virtual identity vectors and the N second virtual identity vectors includes: for an ith second virtual identity vector among the N second virtual identity vectors (i is a positive integer from 1 to N), updating a virtual identity vector corresponding to the first target in an ith dynamic list using the ith second virtual identity vector, the ith dynamic list including virtual identity vectors of different targets at the ith resolution; determining an identity sub-loss corresponding to the i-th first virtual identity vector based on the i-th first virtual identity vector and the updated i-th dynamic list; and determining as the identity loss a sum of identity sub-losses corresponding to the N first virtual identity vectors, respectively.

6. Determining an identity sub-loss corresponding to the i-th first virtual identity vector based on the i-th first virtual identity vector and the updated i-th dynamic list includes: obtaining a first ratio between the i-th second virtual identity vector and a first predetermined value, multiplying the first ratio by the i-th first virtual identity vector to obtain a first result, and performing an exponentiation operation on the first result to obtain a first operation value; obtaining a second ratio between each second virtual identity vector in the updated i-th dynamic list and a first predetermined value, and for each second ratio, multiplying the second ratio by a corresponding i-th first virtual identity vector to obtain a second result, and performing an exponentiation operation on the second result to obtain a second operation value corresponding to each second virtual identity vector; determining a sum of second operations corresponding to each second virtual identity vector, obtaining a third ratio of the first operations to the sum, and performing a logarithmic operation on the third ratio to obtain a third operation; and determining a negative value of the third calculation value as an identity sub-loss corresponding to the i-th first virtual identity vector.

7. The attribute module includes an encoding unit and a decoding unit, and the step of performing attribute vector extraction on the second training image by the attribute module in the target network model to obtain M attribute vectors (M is a positive integer) is: performing feature extraction on the second training image by the encoding unit to obtain feature information of the second training image; The method of any one of claims 1 to 6, further comprising: decoding the feature information by the decoding unit to obtain M attribute vectors.

8. The fusion module includes a plurality of different resolution layers, and the step of performing image generation based on the N first virtual identity vectors and the M attribute vectors by the fusion module in the target network model to obtain an identity anonymized image of the second training image includes:

7. The method for training a model according to claim 1, further comprising: based on a resolution corresponding to the N first virtual identity vectors, using the N first virtual identity vectors as patterns and the M attribute vectors as noises to input into a corresponding resolution layer to obtain an identity anonymized image of the second training image.

9. determining a loss for the target network model based on the identity anonymized image, the divergence constraint, and the identity loss, determining a reconstruction loss between the identity anonymized image and the second training image; and determining a loss for the target network model based on the reconstruction loss, the divergence constraint, and the identity loss.

10. The model training method includes: determining a first distance between the identity anonymized image and the first training image, a second distance between the identity anonymized image and the second training image, and a third distance between the first training image and the second training image; determining a contrast loss based on the first distance, the second distance, and the third distance; determining a loss for the target network model based on the reconstruction loss, the divergence constraint, and the identity loss, The method of claim 9 , further comprising determining a loss for the target network model based on the reconstruction loss, the divergence constraint, the identity loss, and the contrast loss.

11. Determining a contrast loss based on the first distance, the second distance, and the third distance comprises: determining the sum of the square of the difference between the second distance and the third distance and the first distance; and determining the difference between a predetermined value and the sum as the contrast loss.

12. When the fusion module is an adversarial network, determining a loss for the target network model based on the reconstruction loss, the divergence constraint, the identity loss, and the contrast loss includes: determining an adversarial loss based on the identity anonymized images and the first training images; and determining a weighted sum of the reconstruction loss, the divergence constraint, the identity loss, the contrast loss, and the adversarial loss as a loss for the target network model.

13. 1. A method for identity anonymization performed by a computing device, comprising: Sampling in a first space of a target space of a projection module in a target network model to obtain N identity latent vectors, where N is a positive integer; projecting the N identity latent vectors into a second space of the target space of the projection module in the target network model by a second projection unit of the projection module to obtain N virtual identity vectors; performing attribute vector extraction for the processing target image by an attribute module in a target network model to obtain M attribute vectors (M is a positive integer) of the processing target image; and performing image generation based on the N virtual identity vectors and the M attribute vectors by a fusion module in the target network model to obtain an identity-anonymized image of the processing target image.

14. The mean and variance of the first space satisfy a standard Gaussian distribution, and the step of sampling in the first space to obtain N identity latent vectors comprises: The method of claim 13 , comprising sampling the first space based on a mean and a variance to obtain the N identity latent vectors.

15. A model training device configured to perform the method of claim 1.

16. An identity anonymisation device configured to perform the method according to claim 13.

17. A memory for storing a computer program; A computing device comprising: a processor for executing the computer program to perform the model training method of claim 1.

18. A memory for storing a computer program; A processor configured to execute the computer program to perform the identity anonymization method of claim 13.

19. A computer-readable storage medium storing a computer program for causing a computer to execute the model training method according to claim 1.

20. A computer-readable storage medium storing a computer program for causing a computer to execute the identity anonymization method described in claim 13.

21. A computer program product which, when executed by a processor, causes the processor to perform the identity anonymization method of claim 13.

22. A computer program product which, when executed by a processor, causes the processor to perform the model training method of claim 1.

Citation Information

Patent Citations

  • Face anonymization system and method, and terminal

    CN113642409A

  • Method for learning and testing user learning network to be used for recognizing obfuscated data created by concealing original data to protect personal information and learning device and testing device using the same

    WO2021080103A1