Model training and identity anonymization method, device, equipment and storage medium

Abstract

The present application provides a model training and identity anonymization method, device, equipment and storage medium, which can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method includes: sampling on the target space of the projection module in the target network model to obtain N virtual identity vectors; processing the image to be processed through the attribute module in the target network model to obtain M attribute vectors; inputting the N virtual identity vectors and the M attribute vectors into the fusion module of the target network model to obtain the identity anonymization image of the image to be processed. That is, when the identity is anonymized, the present application generates an independent virtual identity through the target network model, without removing the facial area in the image, thereby improving the fidelity and resolution of the identity anonymization.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a model training and identity anonymization method, apparatus, device, and storage medium. Background Art

[0002] Identity anonymization, also known as de-identification, refers to the removal of identifiable identity features (identity) from images or videos, while retaining other non-identity-related attributes unchanged and ensuring that the anonymized images or videos must still be visually authentic.

[0003] Currently, anonymized images are generated using conditional generative adversarial networks (GANs). Specifically, pose keypoints are extracted from the original image and then fed into the model along with the background image after removing the facial region. This input is used to generate a new virtual identity to fill the missing facial region. However, this method uses the background image after removing the facial region as input, resulting in poor quality images. Summary of the Invention

[0004] The present application provides a model training and identity anonymization method, apparatus, device and storage medium to improve the generation quality of identity anonymized images.

[0005] In a first aspect, the present application provides a model training method, comprising:

[0006] Projecting the first training image into the target space through a projection module in the target network model to obtain N first virtual identity vectors, where N is a positive integer;

[0007] Processing the second training image by the attribute module in the target network model to obtain M attribute vectors, where M is a positive integer;

[0008] Inputting the N first virtual identity vectors and the M attribute vectors into a fusion module of the target network model to obtain an identity anonymized image of the second training image;

[0009] The loss of the target network model is determined based on the identity anonymized image, and the target network model is trained based on the loss.

[0010] In a second aspect, a method for anonymizing an identity is provided, comprising:

[0011] Sampling is performed on the target space of the projection module in the target network model to obtain N virtual identity vectors, where N is a positive integer;

[0012] The image to be processed is processed by the attribute module in the target network model to obtain M attribute vectors, where M is a positive integer;

[0013] The N virtual identity vectors and the M attribute vectors are input into a fusion module of the target network model to obtain an identity anonymized image of the image to be processed.

[0014] In a third aspect, a model training device is provided, comprising:

[0015] a projection unit, configured to project the first training image into a target space through a projection module in a target network model to obtain N first virtual identity vectors, where N is a positive integer;

[0016] an attribute unit, configured to process the second training image through the attribute module in the target network model to obtain M attribute vectors, where M is a positive integer;

[0017] a fusion unit, configured to input the N first virtual identity vectors and the M attribute vectors into a fusion module of the target network model to obtain an identity anonymized image of the second training image;

[0018] A training unit is configured to determine a loss of the target network model based on the identity anonymized image, and train the target network model based on the loss.

[0019] In a fourth aspect, an identity anonymization device is provided, comprising:

[0020] A sampling unit, configured to perform sampling on a target space of a projection module in a target network model to obtain N virtual identity vectors, where N is a positive integer;

[0021] An attribute unit, configured to process the image to be processed by an attribute module in a target network model to obtain M attribute vectors, where M is a positive integer;

[0022] The anonymization unit is used to input the N virtual identity vectors and the M attribute vectors into the fusion module of the target network model to obtain the identity anonymized image of the image to be processed.

[0023] In a fifth aspect, a computing device is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to execute the method of the first aspect or the second aspect and its respective implementations.

[0024] In a sixth aspect, a chip is provided for implementing the method of any one of the first and second aspects above, or their respective implementations. Specifically, the chip includes a processor for loading and executing a computer program from a memory, causing a device equipped with the chip to perform the method of any one of the first or second aspects above, or their respective implementations.

[0025] In a seventh aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method of any one of the above-mentioned first or second aspects or their respective implementations.

[0026] In an eighth aspect, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions enable a computer to execute the method of any one of the first or second aspects or their respective implementations.

[0027] In a ninth aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method in any one of the first or second aspects or their respective implementations.

[0028] In summary, the present application projects the first training image to the target space through the projection module in the target network model to obtain N first virtual identity vectors, processes the second training image through the attribute module in the target network model to obtain M attribute vectors, inputs the N first virtual identity vectors and the M attribute vectors into the fusion module of the target network model to obtain the identity anonymized image of the second training image, determines the loss of the target network model based on the identity anonymized image, and trains the target network model based on the loss. That is, the present application provides a new target network model, which enables the target network model to fully learn the identity information in the first training image and the attribute information in the second training image through the above-mentioned training method. In this way, when the identity is anonymous, sampling is performed on the target space of the projection module in the target network model to obtain N virtual identity vectors; the image to be processed is processed through the attribute module in the target network model to obtain M attribute vectors; the N virtual identity vectors and the M attribute vectors are input into the fusion module of the target network model to obtain the identity anonymized image of the image to be processed. That is, when anonymizing an identity, this application generates an independent virtual identity through a target network model without removing the facial area in the image, thereby improving the fidelity and resolution of the identity anonymization. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1A Schematic diagram of a real image;

[0031] Figure 1B-1D for Figure 1A Schematic diagram of the corresponding identity anonymized image;

[0032] Figure 2 A schematic diagram of a system architecture involved in an embodiment of the present application;

[0033] Figure 3 A flowchart of a model training method provided in one embodiment of the present application;

[0034] Figure 4 A schematic diagram of the structure of a target network model provided in one embodiment of the present application;

[0035] Figure 5 A schematic diagram of the structure of a target network model provided in another embodiment of the present application;

[0036] Figure 6 A schematic diagram of the structure of a target network model provided in another embodiment of the present application;

[0037] Figure 7 This is a schematic diagram of the fusion module structure involved in the embodiment of this application;

[0038] Figure 8 A schematic diagram of the structure of a target network model provided in another embodiment of the present application;

[0039] Figure 9 A schematic diagram is provided for determining the contrast loss involved in the embodiment of the present application;

[0040] Figure 10 Another schematic diagram is provided for determining the contrast loss involved in the embodiment of the present application;

[0041] Figure 11 A flowchart of an identity anonymization method provided in one embodiment of the present application;

[0042] Figure 12 A schematic diagram of a projection module provided in an embodiment of the present application;

[0043] Figure 13 This is a schematic diagram of determining the identity anonymization image involved in the embodiment of this application;

[0044] Figure 14 is a schematic block diagram of a model training device provided in one embodiment of the present application;

[0045] Figure 15 is a schematic block diagram of an identity anonymization device provided in one embodiment of the present application;

[0046] Figure 16 It is a schematic block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0048] It should be understood that in the embodiments of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0049] In the description of the present application, unless otherwise specified, "plurality" means two or more than two.

[0050] In addition, to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0051] To facilitate understanding of the embodiments of the present application, the following briefly introduces the relevant concepts involved in the embodiments of the present application:

[0052] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0053] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0054] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0055] The method of the embodiment of the present application can be applied to any scenario where an image needs to be anonymized. For example, Figures 1A to 1D As shown, Figure 1A is a real image, Figures 1B to 1D for Figure 1A Anonymized image of the identity. Figure 1A and Figures 1B to 1D It can be seen that Figures 1B to 1D Removal Figure 1A The identifiable identity features (Identity) are retained while other attributes not related to the identity remain unchanged and ensure that they are still visually authentic.

[0056] Scenario 1: The embodiments of this application can be applied to privacy protection scenarios. For example, for images or videos related to faces, the method of the embodiments of this application can be used to replace the real identity with a virtual identity, so that subsequent detection tasks can continue to be performed without leaking privacy. In addition, users can also use the method of the embodiments of this application to hide their identity characteristics when posting images or videos to avoid the leakage of real information.

[0057] Scenario 2: The embodiments of the present application can be applied to generate virtual image scenarios. For example, the technical solutions of the embodiments of the present application can be used to generate virtual identities, such as fixing identity hidden variables, replacing background images, and generating pictures or videos of a specific virtual image in different scenarios.

[0058] It should be noted that the above-mentioned scenarios 1 and 2 are described using human faces as an example. The method of the embodiments of the present application can also be applied to scenarios where the identities of other non-human faces are anonymized, such as anonymizing the identities of any targets such as animals and vehicles in the processed image.

[0059] In some embodiments, the method of the embodiment of the present application can be applied to an intelligent transportation system. The intelligent transportation system (ITS), also known as the intelligent transportation system (ITS), is an effective and comprehensive application of advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthening the connection between vehicles, roads and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy. Exemplarily, the solution of combining the present application with intelligent transportation can be that the vehicle-mounted equipment collects the user's facial image, and uses the method of the embodiment of the present application to anonymize the identity of the collected facial image, and then sends it to other devices for task analysis, such as illegal driving analysis or intelligent driving analysis.

[0060] Figure 2 This is a schematic diagram of a system architecture involved in an embodiment of the present application, including a user device 101, a data acquisition device 102, a training device 103, an execution device 104, a database 105, a content library 106, an I / O interface 107 and a target network model 108.

[0061] The data acquisition device 102 is used to read training data from the content library 106 and store the read training data in the database 105. The training data involved in the embodiment of the present application includes a first training image, a second training image, and a third training image, and the first training image, the second training image, and the third training image are all used to train the target network model.

[0062] In some embodiments, the user device 101 is used to perform labeling operations on the data in the database 105 .

[0063] The training device 103 trains the target network model 108 based on the training data maintained in the database 105, so that the trained target network model 108 can generate an identity-anonymized image of the image to be processed. Optionally, the target network model 108 obtained by the training device 103 can be applied to different systems or devices.

[0064] In the attached Figure 2In the example, execution device 104 is equipped with an I / O interface 107 for data exchange with external devices. For example, it receives an image to be processed from user device 101 via the I / O interface. The computing module 109 in execution device 104 processes the input image to be processed using the trained target network model 108, outputting an identity-anonymized image. The generated identity-anonymized image is then sent to user device 101 for display or input into other task models for further processing.

[0065] The user device 101 may include a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile internet device (MID), or other terminal devices with a browser installed.

[0066] The execution device 104 can be a server. The server can be one or more. When there are multiple servers, at least two servers are used to provide different services, and / or at least two servers are used to provide the same service, such as providing the same service in a load balancing manner. This is not limited in the embodiments of the present application. The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also become a node in the blockchain.

[0067] In this embodiment, the execution device 104 is connected to the user device 101 via a network. The network can be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, or a call network.

[0068] It should be noted that the attached Figure 2This is merely a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. In some embodiments, the data acquisition device 102, user device 101, training device 103, and execution device 104 can be the same device. The database 105 can be distributed on a single server or multiple servers, and the content library 106 can be distributed on a single server or multiple servers.

[0069] The following describes the technical solutions of the embodiments of the present application in detail through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0070] This application provides a target network model for performing identity anonymization processing on a target (e.g., a face) in an image to be processed, thereby generating an identity-anonymized image of the image to be processed. Therefore, in some embodiments, the target network model can be referred to as an identity anonymization model or an identity anonymizer.

[0071] First, the training process of the target network model is introduced.

[0072] Figure 3 This is a flowchart of a model training method provided in one embodiment of the present application. The execution subject of this embodiment of the present application is a device with a model training function, such as a model training device, which can be a computing device or a portion of a computing device. The following description uses the computing device as an example.

[0073] like Figure 3 As shown, the method of the embodiment of the present application includes:

[0074] S301. Project the first training image to the target space through the projection module in the target network model to obtain N first virtual identity vectors, where N is a positive integer.

[0075] The first training image in the embodiment of the present application is a training image in the training data. It should be noted that if the first training image is a face image, the first training image is obtained with the user's permission.

[0076] In the embodiment of the present application, the process of training the model with each first training image is basically similar. For the convenience of description, a first training image is used as an example for explanation.

[0077] In this embodiment, a target network model is used to project a first training image into a target space, thereby obtaining one or more virtual identity vectors for the first training image. This allows the target network model to learn the identity information of the first training image. After the target network model has fully learned the identity information, when performing identity anonymization, the target space of the target network model can be directly sampled to generate virtual identity vectors.

[0078] The embodiments of the present application mainly involve the concepts of attribute vectors and virtual identity vectors.

[0079] The virtual identity vector is a vector corresponding to the virtual identity information, and the virtual identity information is identity information after hiding identifiable identity features, such as facial information after hiding identifiable facial features.

[0080] The attribute vector is a vector corresponding to the attribute information. Other feature information in the image except the identifiable identity features is called attribute information, such as background information.

[0081] The target network model of the embodiment of the present application can generate an independent virtual identity vector.

[0082] Figure 4 A schematic diagram of the target network model provided in one embodiment of the present application is shown in FIG. Figure 4 As shown, the target network model of the embodiment of the present application includes a projection module, an attribute module and a fusion module.

[0083] The projection module is used to project the first training image into the target space to obtain N first virtual identity vectors of the first training image. The embodiment of the present application does not limit the specific value of N, which is set according to actual needs.

[0084] The attribute module is used to process the second training image to extract M attribute vectors of the second training image. The embodiment of the present application does not limit the specific value of M, which is set according to actual needs. Optionally, M is equal to N.

[0085] The fusion module is used to fuse the N first virtual identity vectors and the M attribute vectors to obtain an identity anonymized image of the second training image.

[0086] If the above N is a positive integer greater than 1, the N first virtual identity vectors correspond to different resolutions respectively.

[0087] As can be seen from the above, in the target network model of the embodiment of the present application, the projection module is used to generate a virtual identity vector for the target in the second training image, wherein this virtual identity vector conceals the target's true identity characteristics in the second training image. The attribute module is used to generate an attribute vector for the second training image, wherein this attribute vector retains features other than the target's true identity characteristics in the second training image. Thus, after the fusion module fuses the virtual identity vector and the attribute vector, it can generate an anonymized image that conceals the target's identity in the second training image, i.e., an identity-anonymized image.

[0088] In some embodiments, as Figure 5 As shown, the projection module includes a first projection unit and a second projection unit, and the target space includes a first space Z and a second space W. In this case, the above S301 includes the following steps S301-A1 to S301-A3:

[0089] S301-A1, extracting prior identity information of a first training image;

[0090] S301-A2, projecting the prior identity information into a first space Z through a first projection unit to obtain N identity latent vectors;

[0091] S301-A3. Project the N identity latent vectors into the second space W through the second projection unit to obtain N first virtual identity vectors.

[0092] like Figure 5 As shown, prior identity information of the first training image is first extracted, for example, using a pre-trained recognition model. Next, a first projection unit projects the prior identity information of the first training image into a first space Z to obtain N identity latent vectors. A second projection unit then projects these N identity latent vectors into a second space W to obtain N first virtual identity vectors.

[0093] The first space Z and the second space W are different latent spaces. The embodiment of the present application does not limit the first space Z and the second space W.

[0094] In some embodiments, the first space is a latent space Z, and the latent space Z conforms to a standard Gaussian distribution.

[0095] At this time, the above S301-A2 includes the following steps S301-A21 and S301-A22:

[0096] S301-A21. Project the prior identity information into the mean and variance of the first space by a first projection unit;

[0097] S301-A22: Sampling is performed based on the mean and variance of the first space to obtain N identity latent vectors.

[0098] The embodiment of the present application does not limit the specific network structure of the first projection unit.

[0099] In one possible implementation, the first projection unit is a variational autoencoder (VAE), such as a conditional variational autoencoder (CVAE). A CVAE is a generative network that uses an encoder to learn the distribution of data to generate latent variables, and then uses a decoder to restore the latent variables to the original form of the data. CVAEs can learn the data distribution and then sample new data, and are commonly used for image generation.

[0100] In this way, the prior identity information of the first training image can be input into the VAE, which projects the prior identity information into the mean and variance of the first space. Then, sampling is performed based on the mean and variance of the first space to obtain N identity latent vectors of the first training image.

[0101] In this example, the first space is a latent space Z that conforms to the standard Gaussian distribution. Therefore, in order to further enhance the expressive power of the latent space, the embodiment of the present application generates different latent vectors at different resolution levels, for example, generating N identity latent vectors, which is equivalent to constructing a latent space containing multiple identity latent vectors. space.

[0102] In some embodiments, the second space W is obtained from the latent space Z, for example, by performing a linear or nonlinear mapping on the latent space Z.

[0103] The embodiment of the present application does not limit the specific network structure of the second projection unit, for example, it is a mapping network composed of multiple fully connected layers.

[0104] In an embodiment of the present application, the prior identity information of the first training image is projected into the shadow space (i.e., the target space) of the projection module, so that the projection module can fully learn the identity information of the first training image, so as to subsequently generate a virtual identity vector that conforms to reality.

[0105] S302: Process the second training image through the attribute module in the target network model to obtain M attribute vectors, where M is a positive integer.

[0106] The second training image is any image in the training data set, and the second training image and the first training image may be the same image or different images.

[0107] The attribute module of the embodiment of the present application is used to learn the attribute information of the second training image to generate M attribute vectors.

[0108] The embodiment of the present application does not limit the specific network model of the attribute module.

[0109] In one possible implementation, Figure 6 As shown, the attribute module includes an encoding unit and a decoding unit. In this case, the above S302 includes the following steps:

[0110] S302-A1, inputting the second training image into the encoding unit to obtain feature information of the second training image;

[0111] S302-A2: Input the feature information into the decoding unit to obtain M attribute vectors.

[0112] In some embodiments, the encoding unit includes multiple feature extraction layers, the decoding unit also includes multiple feature extraction units, and at least one feature extraction layer in the encoding unit is jump-connected to at least one feature extraction layer in the decoding unit.

[0113] After generating N first virtual identity vectors and M attribute vectors according to the above steps, the following S303 is executed.

[0114] S303 : Input the N first virtual identity vectors and the M attribute vectors into a fusion module of a target network model to obtain an identity anonymized image of the second training image.

[0115] In the embodiment of the present application, the specific method of inputting N first virtual identity vectors and M attribute vectors into the fusion module of the target network model to obtain the identity anonymized image of the second training image is not limited.

[0116] Example 1: N first virtual identity vectors are spliced, and M attribute vectors are spliced ​​at the same time. The spliced ​​first virtual identity vectors and attribute vectors are fused and input into the fusion module.

[0117] For example, the spliced ​​first virtual identity vector and the attribute vector are concatenated and then input into the fusion module.

[0118] For another example, the concatenated first virtual identity vector and attribute vector are added together and then input into the fusion module.

[0119] Example 2: The fusion module includes multiple layers with different resolutions. In this case, the above S303 includes the following steps:

[0120] S303-A: Based on the resolutions corresponding to the N first virtual identity vectors, the N first virtual identity vectors are used as patterns and the M attribute vectors are used as noises, and are input into the corresponding resolution layer to obtain an identity anonymized image of the second training image.

[0121] For example, N is 3 and M is 4. The fusion module includes four different resolution layers, where the three first virtual identity vectors are recorded as first virtual identity vector 1, first virtual identity vector 2, and first virtual identity vector 3, and the four attribute vectors are recorded as attribute vector 1, attribute vector 2, attribute vector 3, and attribute vector 4. The four resolution layers are recorded as resolution layer 1, resolution layer 2, resolution layer 3, and resolution layer 4 in order of resolution. The first virtual identity vector 1 corresponds to resolution layer 1 and resolution layer 2 with lower resolution, the first virtual identity vector 2 corresponds to resolution layer 3 with medium resolution, and the virtual identity vector 3 corresponds to resolution layer 4 with the highest resolution. The four attribute vectors correspond to the four resolution layers in order of resolution.

[0122] Exemplarily, the first virtual identity vector 1 is input into resolution layer 1 to obtain feature information 1. Attribute vector 1 is merged with feature information 1 and then input into resolution layer 2 along with the first virtual identity vector 1 to obtain feature information 2. Attribute vector 2 is merged with feature information 2 and then input into resolution layer 3 along with the first virtual identity vector 3 to obtain feature information 3. Attribute vector 3 is merged with feature information 3 and then input into resolution layer 4 along with the first virtual identity vector 4 to obtain feature information 4. Finally, feature information 4 and attribute vector 4 are merged and processed to generate an identity-anonymized image of the second training image.

[0123] The embodiment of the present application does not limit the specific network structure of the fusion module.

[0124] In one possible implementation, the fusion module is StyleGAN2 (Style-based generator). Figure 7 As shown, an AdaIN layer is included between two adjacent resolution layers in the fusion module. For example, an affine transform (AT) is performed on the first virtual identity vector i+1, and the feature information i outputted by the i-th resolution layer is merged with the attribute vector i. The combined information and the affine transformed first virtual identity vector i+1 are input into the AdaIN layer, and the AdaIN operation is performed. The result of the AdaIN operation is then input into the i+1-th resolution layer.

[0125] The fusion module of the embodiment of the present application can also be an adversarial model such as StyleGAN3 and ProGAN. When the fusion module adopts different adversarial models, the method of determining the identity anonymization image of the second training image may be different. The embodiment of the present application does not limit this and is determined based on actual conditions.

[0126] In a specific embodiment, taking the first projection unit as VAE, the second projection unit as the mapping network, the attribute module as the autoencoder, and the fusion module as StyleGAN2 as an example, the model training process of the embodiment of the present application is introduced.

[0127] Specifically, such as Figure 8 As shown, the first training image Xs is passed through a pre-trained face recognition model to generate prior identity information. Next, the prior identity information is input into the VAE, which projects the prior identity information into the first space Z through the VAE to obtain N identity latent vectors. For example, 3 N identity latent vectors are obtained, and these 3 N identity latent vectors correspond to three different resolutions: low, medium, and high. Next, the N identity latent vectors are input into the mapping network, which projects the N identity latent vectors from the first space Z to the second space W through the mapping network to obtain N first virtual identity vectors. In addition, the second training image Xt is input into the autoencoder, and after the second training image Xt is processed by the autoencoder, M attribute vectors are generated. Finally, the M attribute vectors are used as noise, and the N first virtual identity vectors are used as styles, and are input into each layer of StyleGAN2 to obtain the identity anonymized image Ys,t of the second training image output by StyleGAN2.

[0128] According to the above steps, the first training image and the second training image are input into the target network model to obtain the identity anonymized image of the second training image output by the target network model. Then, the following S304 is executed to train the target network model.

[0129] S304: Determine the loss of the target network model based on the identity anonymized image, and train the target network model based on the loss.

[0130] According to the above steps, the target network model outputs an identity anonymized image of the second training image, and the loss of the target network model is determined based on the identity anonymized image.

[0131] In the embodiment of the present application, there is no limitation on the specific method for determining the loss of the target network model based on the identity anonymized image.

[0132] In one possible implementation, the identity anonymized image is input into a judgment model, which is a pre-trained model that can predict the degree of anonymization of the identity anonymized image. For example, the identity anonymized image is input into the judgment model, and the judgment model performs identity recognition on the identity anonymized image, and determines the recognition result as the loss of the target network model. If the recognition accuracy is high, it means that the anonymization effect of the current target network model is not ideal. At this time, the parameters in the target network model are adjusted according to the loss of the target network model. Then, a new first training image and a second training image are selected to perform the above steps S301 to S304, and the target network model is continued to be trained until the target network model reaches the training end condition. The training end condition includes at least that the number of training times reaches a preset number of times, or the anonymization degree of the model reaches the expected effect.

[0133] In another possible implementation, if Figure 5 The first space Z shown is a latent space that conforms to the standard Gaussian distribution. In this application, the N identity latent vectors in the first space Z are constrained by KL divergence. , to ensure that the identity information is projected into the standard Gaussian distribution.

[0134] Based on this, the embodiment of the present application further includes: determining the divergence constraints of the N identity latent vectors. In this case, the above S304 includes:

[0135] S304-A. Determine the loss of the target network model based on the identity anonymized image and the divergence constraint.

[0136] For example, the divergence constraint of N identity latent vectors can be determined by the following formula (1): :

[0137] (1)

[0138] Among them, μ i is the mean value corresponding to the i-th identity latent vector among the N identity latent vectors, σ i is the variance corresponding to the i-th identity latent vector among the N identity latent vectors.

[0139] It should be noted that the above formula (1) is only an example. The method of determining the divergence constraints of N identity latent vectors in the embodiment of the present application includes but is not limited to the above formula (1). For example, it can be other methods of calculating the divergence constraints, such as deforming the above formula (1).

[0140] In this embodiment of the application, the divergence constraint of N identity latent vectors After training, not only does the projection module fully learn the identity information, but the first space of the projection module also satisfies the standard Gaussian distribution. In this way, during the subsequent anonymization processing, the first space can be directly sampled to generate N identity latent vectors that conform to the standard Gaussian distribution for generating virtual identity vectors.

[0141] In some embodiments, the second space is obtained by nonlinear mapping from the first space and is a complex non-Gaussian distribution. Figure 5 As shown, after mapping the identity information to the first space, it is found that the intermediate latent space second space The distribution is not uniform, the real identity vectors are concentrated in multiple different centers, and do not overlap with the generated virtual identity vectors, so the virtual identity vectors cannot produce a reasonable face identity. Therefore, the embodiment of the present application proposes to use a contrast loss to The latent vector of the space (i.e., the first virtual identity vector) is constrained so that the latent vectors from the same identity are aggregated together, while repelling the latent vectors of different identities, and all the latent vectors are evenly distributed in the entire space.

[0142] Based on this, the method of the embodiment of the present application further includes the following steps:

[0143] Step 1: Obtain a third training image;

[0144] Step 2: Process the third training image through the projection reference module to obtain N second virtual identity vectors;

[0145] Step 3: Determine identity loss based on the N first virtual identity vectors and the N second virtual identity vectors.

[0146] The third training image and the first training image are two different images of the first target. For example, the third training image and the first training image are two different facial images of the same user.

[0147] The projection reference module has the same network structure as the projection module and is updated according to the projection module. For example, the projection reference module is updated according to the momentum of the projection module, that is, the projection reference module is updated slowly as the projection module is updated.

[0148] For example, the projection reference module can be updated according to the following formula (2):

[0149] Pθ'(t)=(1-Δ)*Pθ'(t-1)+Δ*Pθ(t)(2)

[0150] Wherein, Pθ'(t) is the projection reference module parameter after the t-th update, Pθ'(t-1) is the projection reference module parameter after the t-1-th update, Pθ(t) is the projection module parameter after the t-th update, and Δ is a small value, for example, 0.01.

[0151] like Figure 9 As shown, during the model training process, in order to determine the identity loss, the embodiment of the present application sets a projection reference module that is completely consistent with the network structure of the projection module to constrain the first virtual identity vector output by the projection module. Specifically, the first training image is input into the projection module to obtain N first virtual identity vectors of the first training image, and the third training image is input into the projection reference module to obtain N second virtual identity vectors of the third training image. Since the first training image and the third training image are images of the same target, and the network structure of the projection module is consistent with that of the projection reference module, if the model training is completed, the difference between the N first virtual identity vectors and the N second virtual identity vectors corresponding to the first training image is small. Based on this, the projection module in the target network model can be trained according to the N first virtual identity vectors and the N second virtual identity vectors corresponding to the first training image, so that the projection module can generate a virtual identity vector that meets the requirements.

[0152] In step 1 above, the identity loss is determined based on the N first virtual identity vectors and the N second virtual identity vectors. Methods include but are not limited to the following:

[0153] Method 1: Determine the differences between N first virtual identity vectors and N second virtual identity vectors at different resolutions, and determine the sum or average of the differences as the identity loss. For example, if N is 3, determine the difference 1 between first virtual identity vector 1 and second virtual identity vector 1, the difference 2 between first virtual identity vector 2 and second virtual identity vector 2, and the difference 3 between first virtual identity vector 3 and second virtual identity vector 3. The sum of the differences 1, 2, and 3 is determined as the identity loss, or the average of the differences 1, 2, and 3 is determined as the identity loss.

[0154] Method 2: This embodiment of the application designs N dynamic lists , the dynamic list stores all different target identities (such as face identities) in the entire training set in the second space In this case, the above step 3 includes the following steps 31 to 33:

[0155] Step 31 : For the i-th first virtual identity vector among the N first virtual identity vectors, use the i-th second virtual identity vector to update the virtual identity vector corresponding to the first target in the i-th dynamic list.

[0156] The i-th dynamic list includes virtual identity vectors of different targets at the i-th resolution, where i is a positive integer from 1 to N.

[0157] In an embodiment of the present application, among the N second virtual identity vectors, each of the N second virtual identity vectors corresponds to a dynamic list, for example, N is 3, corresponding to low resolution, medium resolution and high resolution respectively. In this way, the dynamic list also includes 3, namely, a first dynamic list corresponding to low resolution, a second dynamic list corresponding to medium resolution and a third dynamic list corresponding to high resolution.

[0158] Assume that i=1, and use the first second virtual identity vector to update the virtual identity vector corresponding to the first target in the first dynamic list.

[0159] Assume that i=2, and use the second second virtual identity vector to update the virtual identity vector corresponding to the first target in the second dynamic list.

[0160] Assume that i=3, and use the third second virtual identity vector to update the virtual identity vector corresponding to the first target in the third dynamic list.

[0161] Step 32 : Determine the identity sub-loss corresponding to the i-th first virtual identity vector according to the i-th first virtual identity vector and the updated i-th dynamic list.

[0162] For example, Figure 10 As shown, the first training image and the third training image are two different images of the first target j. The first training image Xj is input into the projection module to obtain N first virtual identity vectors Wj, and the third training image Xj' is input into the projection reference module to obtain N second virtual identity vectors Wj'. For the i-th resolution among the N resolutions, the i-th dynamic list Ki includes the second virtual identity vectors of different targets at the i-th resolution, and the i-th dynamic list Ki is updated in real time. Specifically, the i-th second virtual identity vector is used to update the virtual identity vector kj corresponding to the first target j in the i-th dynamic list Ki, that is, kj is updated to Wj'. Then, based on the i-th second virtual identity vector and the updated i-th dynamic list, the identity sub-loss i corresponding to the i-th first virtual identity vector is determined.

[0163] The embodiment of the present application does not limit the specific method of determining the identity sub-loss corresponding to the i-th first virtual identity vector in the above step 32.

[0164] For example, using loss methods such as center loss and triplet loss, the identity sub-loss corresponding to the i-th first virtual identity vector is determined according to the i-th first virtual identity vector and the updated i-th dynamic list.

[0165] In some embodiments, the above step 32 of determining the identity sub-loss corresponding to the i-th first virtual identity vector among the N first virtual identity vectors includes the following steps:

[0166] Step 321: After comparing the i-th second virtual identity vector with a first preset value, multiplying it with the i-th first virtual identity vector and then performing an exponential operation to obtain a first operation value;

[0167] Step 322: Compare each second virtual identity vector in the updated i-th dynamic list with the first preset value, multiply it by the corresponding i-th first virtual identity vector, and then perform an exponential operation to obtain a second operation value corresponding to each second virtual identity vector;

[0168] Step 323: Compare the first operation value with the sum of the second operation values ​​corresponding to each second virtual identity vector, and then perform a logarithmic operation to obtain a third operation value;

[0169] Step 324 : Determine the negative of the third operation value as the identity sub-loss corresponding to the i-th first virtual identity vector.

[0170] For example, As anchor point, The first The items are positive samples, and the rest are negative samples. The identity sub-loss is determined using the contrast loss in the form of InfoNCE (Information Noise Contrastive Noise, information noise contrast estimation) , where InfoNCE is a loss function that modifies autoregression based on mutual information.

[0171] Exemplarily, the identity sub-loss corresponding to the i-th first virtual identity vector is determined according to the following formula (3): :

[0172] (3)

[0173] in, is the first virtual identity vector of the first target j, is the second virtual identity vector of the first target j, is the first preset value, is the i-th second virtual identity vector corresponding to the k-th target in the i-th dynamic list, is the first virtual identity vector corresponding to the k-th target, and K is the total number of targets included in the i-th dynamic list.

[0174] It should be noted that the above formula (3) is only an example.

[0175] Step 33: Determine the sum of the identity sub-losses corresponding to the N first virtual identity vectors as the identity loss of the target network model.

[0176] After determining the identity sub-loss corresponding to the i-th first virtual identity vector in step 32, the identity loss is determined by summing the identity sub-losses corresponding to the N first virtual identity vectors. For example, if N is 3, the identity sub-loss corresponding to each of the three first virtual identity vectors is determined according to the above method, and the identity loss of the model is determined by summing the identity sub-losses corresponding to these three first virtual identity vectors.

[0177] In this embodiment of the present application, after determining the identity loss during the model training process according to the above method, the above S304-A determines the loss of the target network model based on the identity anonymized image and the divergence constraint, including the following steps:

[0178] S304-A1. Determine the loss of the target network model based on the identity anonymized image, the divergence constraint, and the identity loss.

[0179] In some embodiments, the above S304-A1 includes the following steps:

[0180] S304-A11, determining a reconstruction loss between the identity anonymized image and a second training image;

[0181] S304-A12. Determine the loss of the target network model based on the reconstruction loss, the divergence constraint, and the identity loss.

[0182] This embodiment of the present application does not limit the method for determining the reconstruction loss between the identity-anonymized image and the second training image.

[0183] In one example, the difference between the identity-anonymized image and the second training image is determined as the reconstruction loss. For example, the sum of the differences between each pixel of the identity-anonymized image and the corresponding pixel of the second training image is determined as the reconstruction loss.

[0184] In another example, the reconstruction loss is determined according to the following formula (4): :

[0185] (4)

[0186] in, To anonymize images for identity, is the second training image, It is a 1-norm operation.

[0187] According to the above steps, the reconstruction loss is determined Finally, the loss of the target network model is determined based on the reconstruction loss, the divergence constraint, and the identity loss. For example, the weighted sum of the reconstruction loss, the divergence constraint, and the identity loss is determined as the final loss of the target network model.

[0188] In some embodiments, in order to improve the training accuracy of the model, the embodiment of the present application further includes determining the identity contrast loss of the identity anonymized image, specifically, including the following steps:

[0189] Step A: determining a first distance between the identity-anonymized image and the first training image, a second distance between the identity-anonymized image and the second training image, and a third distance between the first training image and the second training image;

[0190] Step B, determining the contrast loss according to the first distance, the second distance, and the third distance;

[0191] The first distance, the second distance and the third distance may be determined by any distance method such as cosine distance.

[0192] Example 1: After determining the first distance, the second distance, and the third distance according to step A, the sum of the first distance, the second distance, and the third distance is determined as the contrast loss.

[0193] Example 2: Determine the sum of the square of the difference between the second distance and the third distance and the first distance; and determine the difference between a preset value and the sum as the contrast loss.

[0194] In one example, the contrast loss is determined according to the following formula (5): :

[0195] (5)

[0196] Among them, z id represents the 512-dimensional identity vector representation of image X extracted from the pre-trained face recognition model, is the first distance between the identity anonymized image and the first training image, is the second distance between the identity anonymized image and the second training image, is the third distance between the first training image and the second training image.

[0197] According to the above steps, the contrast loss is determined Finally, the loss of the target network model is determined according to the reconstruction loss, the divergence constraint, the identity loss and the contrast loss. For example, the weighted sum of the reconstruction loss, the divergence constraint, the identity loss and the contrast loss is determined as the loss of the target network model.

[0198] In some embodiments, if the fusion module is an adversarial network, the adversarial loss of the model is also determined during the model training process. Specifically, the adversarial loss is determined based on the identity anonymized image and the first training image.

[0199] For example, according to the following formula (6), the adversarial loss is determined :

[0200] (6)

[0201] Among them, D is the discriminator, G is the generator, and E(*) represents the expected value of the distribution function. The first training image for the discriminator The identification results, Anonymize images for discriminator identity identification results.

[0202] According to the above steps, determine the adversarial loss Finally, the loss of the target network model can be determined according to the reconstruction loss, divergence constraint, identity loss, contrast loss and adversarial loss. For example, the weighted sum of the reconstruction loss, divergence constraint, identity loss, contrast loss and adversarial loss is determined as the loss of the target network model.

[0203] It should be noted that the embodiments of the present application do not limit the specific sizes of the weight values ​​corresponding to the reconstruction loss, divergence constraint, identity loss, contrast loss and adversarial loss, and they are determined according to actual needs.

[0204] In one possible implementation, the reconstruction loss, divergence constraint, identity loss, contrast loss, and adversarial loss are weighted according to the following formula (7) to obtain the loss of the target network model: :

[0205] (7)

[0206] The weights corresponding to the losses in the above formula (7) are an example. The weights corresponding to the losses in the embodiments of the present application include but are not limited to those shown in the above formula (7), and are determined according to specific needs.

[0207] In some embodiments, in order to further improve the training accuracy of the target network model, other losses in addition to the losses described in the above embodiments may be determined. The embodiments of the present application do not limit this and are determined based on actual needs.

[0208] As can be seen from the above, the embodiment of the present application realizes identity anonymization by generating first virtual identity vectors corresponding to different resolutions, which can improve the resolution of anonymization. For example, The anonymized result of high resolution produces fewer image artifacts and has higher fidelity. In addition, the embodiment of the present application does not rely on key regression models and segmentation models during model training, that is, it does not remove the face area in the image, and retains the posture, details and occlusion in the original image.

[0209] In an embodiment of the present application, a first training image is projected into a target space through a projection module in a target network model to obtain N first virtual identity vectors, a second training image is processed through an attribute module in the target network model to obtain M attribute vectors, the N first virtual identity vectors and the M attribute vectors are input into a fusion module of the target network model to obtain an identity anonymized image of the second training image, the loss of the target network model is determined based on the identity anonymized image, and the target network model is trained based on the loss. That is, the present application provides a new target network model, which, through the above-mentioned training method, enables the target network model to learn the identity information in the first training image, so that the target network model can independently generate a virtual identity, and at the same time, allows the target network model to fully learn the attribute information in the second training image. During the entire learning process, there is no need to remove the facial area in the image, nor is there any need to use real identity information for guidance. The target network model is trained by utilizing the clear supervision target in the face-changing task to improve the fidelity and resolution of the identity anonymization generation of the target network model, so that the trained target network model can generate high-quality identity anonymized images.

[0210] Combined with the above Figures 3 to 10 , describes the model training method of this application in detail, and the following is combined with Figures 11 to 13 , the identity anonymization method of this application is described in detail.

[0211] Figure 11 A flowchart of an identity anonymization method provided in one embodiment of the present application. Figure 11 The identity anonymization method shown is to use the above-trained target network model to perform identity anonymization processing.

[0212] like Figure 11 As shown, the method includes:

[0213] S401. Sampling is performed on the target space of the projection module in the target network model to obtain N virtual identity vectors, where N is a positive integer.

[0214] As can be seen from the above embodiment, the present embodiment uses the first training image to train the projection module, allowing the projection module to fully learn the identity information in the first training image. In this way, in actual use, N virtual identity vectors can be obtained by sampling the target space of the projection module.

[0215] Specific implementations of S401 include but are not limited to the following:

[0216] Method 1: Sample the target space mean and variance of the trained projection module to obtain N virtual identity vectors. For example, randomly sample the variance of the target space and add it to the mean of the target space to obtain a virtual identity vector. Repeat the above steps to obtain N virtual identity vectors.

[0217] In mode 2, the target space includes a first space and a second space, and the target network model includes a second projection unit. In this case, the above S401 includes the following steps:

[0218] S401-A1, sampling in the first space to obtain N identity latent vectors;

[0219] S401-A2: Project the N identity latent vectors into the second space through a second projection unit to obtain N virtual identity vectors.

[0220] In the embodiment of the present application, during the actual anonymization, the first projection unit in the projection module is no longer used, and only the second projection unit in the projection module is used for projection. Figure 12 As shown in Figure 1, sampling is performed in the first space Z that conforms to a standard Gaussian distribution to obtain N identity latent vectors, which are then input into the second projection unit. The second projection unit projects the N identity latent vectors into the W space to obtain N virtual identity vectors. Figure 12 In the example, N is 3 and the second projection unit is a mapping network. However, the projection module of the embodiment of the present application is not limited to Figure 12 shown.

[0221] As can be seen above, the first space is trained using the first training image, so that the variance and mean of the first space conform to a standard Gaussian distribution. Thus, sampling is first performed in the first space to generate N identity latent vectors. Specifically, sampling is performed based on the mean and variance of the first space to obtain N identity latent vectors. Random sampling is then performed within the variance of the first space, and then added to the mean of the first space to obtain an identity latent vector. Repeating these steps will yield N identity latent vectors. Next, the N identity latent vectors are projected into the second space via a second projection unit to obtain N virtual identity vectors.

[0222] In some embodiments, the N virtual identity vectors correspond to different resolutions, for example, N=3, where the first virtual identity vector corresponds to low resolution, the second virtual identity vector corresponds to medium resolution, and the third virtual identity vector corresponds to high resolution.

[0223] According to the above method, after obtaining N virtual identity vectors, the following steps S402 and S403 are performed to obtain an identity anonymized image of the image to be processed.

[0224] S402 , processing the image to be processed by the attribute module in the target network model to obtain M attribute vectors, where M is a positive integer.

[0225] The attribute module in the embodiment of the present application is used to extract attribute information from the image to be processed.

[0226] The embodiment of the present application does not limit the network structure of the attribute module.

[0227] In some embodiments, the attribute module includes an encoding unit and a decoding unit. In this case, the above S402 includes the following steps:

[0228] S402-A1, inputting the image to be processed into the encoding unit to obtain feature information of the image to be processed;

[0229] S402-A2: Input the feature information into the decoding unit to obtain M attribute vectors.

[0230] The encoding unit includes multiple feature extraction layers.

[0231] Similarly, the above decoding unit also includes multiple feature extraction layers.

[0232] The feature extraction layer may include a convolutional layer, etc.

[0233] In some embodiments, at least one feature extraction layer in the encoding unit is skip-connected to at least one feature extraction layer in the decoding unit.

[0234] The M attribute vectors generated above may correspond to different resolutions.

[0235] In some embodiments, the target network model is an autoencoder.

[0236] S403 : Input the N virtual identity vectors and the M attribute vectors into the fusion module of the target network model to obtain an identity anonymized image of the image to be processed.

[0237] According to the above steps, N virtual identity vectors and M attribute vectors are generated and input into the fusion module to obtain the identity anonymized image of the image to be processed.

[0238] The implementation of S403 includes but is not limited to the following examples:

[0239] Example 1: N virtual identity vectors are concatenated, and M attribute vectors are concatenated at the same time. The concatenated virtual identity vectors and attribute vectors are fused and input into the fusion module.

[0240] For example, the concatenated virtual identity vector and attribute vector are concatenated and then input into the fusion module.

[0241] For another example, the concatenated virtual identity vector and attribute vector are added together and then input into the fusion module.

[0242] Example 2: The fusion module includes multiple different resolution layers. In this case, according to the resolutions corresponding to the N virtual identity vectors, the N virtual identity vectors can be used as styles and the M attribute vectors as noise, and input into the corresponding resolution layer to obtain the identity anonymized image of the image to be processed.

[0243] The embodiment of the present application does not limit the specific network structure of the fusion module.

[0244] In one possible implementation, the fusion module is StyleGAN2. Figure 7 As shown, an AdaIN layer is included between two adjacent resolution layers in the fusion module. For example, an affine transformation is performed on the virtual identity vector i+1, and the feature information i outputted from the i-th resolution layer is merged with the attribute vector i, and then inputted into the AdaIN layer together with the affine-transformed virtual identity vector i+1, an AdaIN operation is performed, and the result of the AdaIN operation is inputted into the i+1-th resolution layer.

[0245] The fusion module of the embodiment of the present application can also be an adversarial model such as StyleGAN3 and ProGAN.

[0246] In a specific embodiment, taking the second projection unit as a mapping network, the attribute module as an autoencoder, and the fusion module as StyleGAN2 as an example, the identity anonymization process of the embodiment of the present application is introduced.

[0247] Specifically, such as Figure 13 As shown, sampling is performed in the first space Z of the projection module to obtain N identity latent vectors. For example, 3 N identity latent vectors are obtained, corresponding to three different resolutions: low, medium, and high. Next, the N identity latent vectors are input into the mapping network, which projects the N identity latent vectors from the first space Z to the second space W to obtain N virtual identity vectors. In addition, the image to be processed Xt is input into the autoencoder, which processes the image to be processed Xt to generate M attribute vectors. Finally, the M attribute vectors are used as noise and the N virtual identity vectors are used as style, which are input into each layer of StyleGAN2 to obtain the identity anonymized image Ys,t of the image to be processed as output by StyleGAN2.

[0248] The identity anonymization method provided in the embodiments of the present application samples the target space of the projection module in the target network model to obtain N virtual identity vectors. The attribute module in the target network model processes the image to be processed to obtain M attribute vectors. The N virtual identity vectors and the M attribute vectors are input into the fusion module of the target network model to obtain the identity anonymized image of the image to be processed. In other words, the target network model in the embodiments of the present application can independently generate virtual identities. When performing identity anonymization on the image to be processed, there is no need to remove the facial area in the image to be processed, thereby improving the fidelity of the identity anonymization.

[0249] Combined with the above Figures 3 to 13 , describes the method embodiment of the present application in detail, and the following is combined with Figures 14 and 15 , describe in detail the device embodiments of the present application.

[0250] Figure 14 1 is a schematic block diagram of a model training device provided in an embodiment of the present application. The training device 10 may be a computing device or a part of a computing device.

[0251] like Figure 14 As shown, the model training device 10 includes:

[0252] A projection unit 11 is configured to project the first training image into a target space through a projection module in a target network model to obtain N first virtual identity vectors, where N is a positive integer;

[0253] an attribute unit 12, configured to process the second training image through the attribute module in the target network model to obtain M attribute vectors, where M is a positive integer;

[0254] a fusion unit 13, configured to input the N first virtual identity vectors and the M attribute vectors into a fusion module of the target network model to obtain an identity anonymized image of the second training image;

[0255] The training unit 14 is configured to determine a loss of the target network model based on the identity anonymized image, and train the target network model based on the loss.

[0256] In some embodiments, the projection module includes a first projection unit and a second projection unit, the target space includes a first space and a second space, and the projection unit 11 is specifically used to extract the prior identity information of the first training image; through the first projection unit, the prior identity information is projected to the first space to obtain N identity latent vectors; through the second projection unit, the N identity latent vectors are projected to the second space to obtain the N first virtual identity vectors.

[0257] In some embodiments, the projection unit 11 is specifically configured to project the prior identity information into the mean and variance of the first space through the first projection unit; and perform sampling based on the mean and variance of the first space to obtain the N identity latent vectors.

[0258] In some embodiments, the training unit 14 is further configured to determine a divergence constraint of the N identity latent vectors; and determine a loss of the target network model based on the identity anonymized image and the divergence constraint.

[0259] In some embodiments, the N first virtual identity vectors correspond to different resolutions respectively.

[0260] Optionally, the first projection unit is a variational autoencoder.

[0261] In some embodiments, the training unit 14 is further used to obtain a third training image, where the third training image and the first training image are two different images of the first target; the third training image is processed by a projection reference module to obtain N second virtual identity vectors, and the projection reference module has the same network structure as the projection module and is updated according to the projection module; the identity loss is determined based on the N first virtual identity vectors and the N second virtual identity vectors; and the loss of the target network model is determined based on the identity anonymization image, the divergence constraint and the identity loss.

[0262] In some embodiments, the training unit 14 is specifically used to update the virtual identity vector corresponding to the first target in the i-th dynamic list using the i-th second virtual identity vector among the N second virtual identity vectors, wherein the i-th dynamic list includes virtual identity vectors of different targets at the i-th resolution, and i is a positive integer from 1 to N; determine the identity sub-loss corresponding to the i-th first virtual identity vector based on the i-th first virtual identity vector and the updated i-th dynamic list; and determine the sum of the identity sub-losses corresponding to the N first virtual identity vectors as the identity loss.

[0263] In some embodiments, the training unit 14 is specifically used to compare the i-th second virtual identity vector with the first preset value, multiply it with the i-th first virtual identity vector, and then perform an exponential operation to obtain a first operation value; compare each second virtual identity vector in the updated i-th dynamic list with the first preset value, multiply it with the corresponding i-th first virtual identity vector, and then perform an exponential operation to obtain a second operation value corresponding to each second virtual identity vector; compare the first operation value with the sum of the second operation values ​​corresponding to each second virtual identity vector, and then perform a logarithmic operation to obtain a third operation value; and determine the negative of the third operation value as the identity sub-loss corresponding to the i-th first virtual identity vector.

[0264] In some embodiments, the attribute module includes an encoding unit and a decoding unit, and the attribute unit 12 is specifically used to input the second training image into the encoding unit to obtain feature information of the second training image; and input the feature information into the decoding unit to obtain M attribute vectors.

[0265] In some embodiments, at least one feature extraction layer in the encoding unit is jump-connected to at least one feature extraction layer in the decoding unit.

[0266] In some embodiments, the fusion module includes multiple different resolution layers, and the fusion unit 13 is specifically used to input the N first virtual identity vectors as styles and the M attribute vectors as noise into the corresponding resolution layer according to the resolutions corresponding to the N first virtual identity vectors to obtain the identity anonymized image of the second training image.

[0267] In some embodiments, the training unit 14 is specifically configured to determine a reconstruction loss between the identity anonymized image and the second training image; and determine a loss of the target network model based on the reconstruction loss, the divergence constraint, and the identity loss.

[0268] In some embodiments, the training unit 14 is further configured to determine a first distance between the identity anonymized image and the first training image, a second distance between the identity anonymized image and the second training image, and a third distance between the first training image and the second training image; determine a contrast loss based on the first distance, the second distance, and the third distance; and determine a loss of the target network model based on the reconstruction loss, the divergence constraint, the identity loss, and the contrast loss.

[0269] In some embodiments, the training unit 14 is further configured to determine a sum of a square of a difference between the second distance and the third distance and the first distance; and determine a difference between a preset value and the sum as the contrast loss.

[0270] In some embodiments, if the fusion module is an adversarial network, the training unit 14 is further used to determine the adversarial loss based on the identity anonymized image and the first training image; and determine the weighted sum of the reconstruction loss, the divergence constraint, the identity loss, the contrast loss and the adversarial loss as the loss of the target network model.

[0271] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 14 The device shown can perform the above Figure 3 The embodiment of the model training method shown, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the method embodiment corresponding to the computing device. For the sake of brevity, they are not repeated here.

[0272] Figure 15 FIG2 is a schematic block diagram of an identity anonymization device provided in an embodiment of the present application. The identity anonymization device 20 may be a computing device or a part of a computing device.

[0273] like Figure 15 As shown, the identity anonymization device 20 includes:

[0274] A sampling unit 21 is configured to perform sampling on the target space of the projection module in the target network model to obtain N virtual identity vectors, where N is a positive integer;

[0275] An attribute unit 22 is configured to process the image to be processed by using an attribute module in a target network model to obtain M attribute vectors, where M is a positive integer;

[0276] The anonymization unit 23 is configured to input the N virtual identity vectors and the M attribute vectors into a fusion module of the target network model to obtain an identity anonymized image of the image to be processed.

[0277] In some embodiments, the target space includes a first space and a second space, and the target network model includes a second projection unit, a sampling unit 21, which is specifically used to perform sampling on the first space to obtain N identity latent vectors; through the second projection unit, the N identity latent vectors are projected to the second space to obtain the N virtual identity vectors.

[0278] In some embodiments, the mean and variance of the first space satisfy a standard Gaussian distribution, and the sampling unit 21 is specifically configured to perform sampling based on the mean and variance of the first space to obtain the N identity latent vectors.

[0279] In some embodiments, the N virtual identity vectors correspond to different resolutions respectively.

[0280] In some embodiments, the attribute module includes an encoding unit and a decoding unit. The attribute unit 22 is specifically configured to input the image to be processed into the encoding unit to obtain feature information of the image to be processed; and input the feature information into the decoding unit to obtain M attribute vectors.

[0281] In some embodiments, at least one feature extraction layer in the encoding unit is jump-connected to at least one feature extraction layer in the decoding unit.

[0282] In some embodiments, the fusion module includes multiple different resolution layers, and the anonymization unit 23 is specifically used to input the N virtual identity vectors as patterns and the M attribute vectors as noise into the corresponding resolution layer according to the resolutions corresponding to the N virtual identity vectors to obtain the identity anonymized image of the image to be processed.

[0283] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 15 The device shown can perform the above Figure 11 The embodiment of the identity anonymization method shown, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the corresponding method embodiment of the computing device, and for the sake of brevity, they are not repeated here.

[0284] The apparatus of the embodiment of the present application is described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0285] Figure 16 It is a schematic block diagram of a computing device provided in an embodiment of the present application, which is used to execute the above method embodiment.

[0286] like Figure 16As shown, the computing device 30 may include:

[0287] The memory 31 and the processor 32 are configured to store a computer program 33 and transmit the program code 33 to the processor 32. In other words, the processor 32 can call and run the computer program 33 from the memory 31 to implement the method in the embodiment of the present application.

[0288] For example, the processor 32 may be configured to execute the above method steps according to instructions in the computer program 33 .

[0289] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0290] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0291] In some embodiments of the present application, the memory 31 includes but is not limited to:

[0292] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0293] In some embodiments of the present application, the computer program 33 may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to implement the method for recording a page provided by the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 33 in the computing device.

[0294] like Figure 16 As shown, the computing device 30 may further include:

[0295] The transceiver 34 may be connected to the processor 32 or the memory 31 .

[0296] The processor 32 may control the transceiver 34 to communicate with other devices. Specifically, the processor 32 may send information or data to other devices or receive information or data sent by other devices. The transceiver 34 may include a transmitter and a receiver. The transceiver 34 may further include one or more antennas.

[0297] It should be understood that the various components in the computing device 30 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0298] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to perform the method of the above-mentioned method embodiment. Alternatively, the present application also provides a computer program product containing instructions. When the computer is executed by the instructions, the computer is enabled to perform the method of the above-mentioned method embodiment.

[0299] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to perform the method of the above method embodiment.

[0300] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are fully or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., digital video disc (DVD)), or semiconductor media (e.g., solid-state drive (SSD)).

[0301] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0302] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0303] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0304] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A model training method, characterized in that: include: Projecting the first training image into the target space through a projection module in the target network model to obtain N first virtual identity vectors, where N is a positive integer; Processing the second training image by the attribute module in the target network model to obtain M attribute vectors, where M is a positive integer; Inputting the N first virtual identity vectors and the M attribute vectors into a fusion module of the target network model to obtain an identity anonymized image of the second training image; Determining a loss of the target network model based on the identity anonymized image, and training the target network model based on the loss; The step of determining the loss of the target network model according to the identity anonymized image includes: Processing a third training image through a projection reference module to obtain N second virtual identity vectors, and determining identity loss based on the N first virtual identity vectors and the N second virtual identity vectors, wherein the third training image and the first training image are two different images of the first target, the projection reference module has the same network structure as the projection module, and is updated based on the projection module; The loss of the target network model is determined according to the identity anonymized image and the identity loss.

2. The method according to claim 1, characterized in that The projection module includes a first projection unit and a second projection unit, the target space includes a first space and a second space, and the first training image is projected into the target space by the projection module in the target network model to obtain N first virtual identity vectors, including: extracting prior identity information of the first training image; Projecting the prior identity information into a first space through the first projection unit to obtain N identity latent vectors; The N identity latent vectors are projected into a second space by the second projection unit to obtain the N first virtual identity vectors.

3. The method according to claim 2, characterized in that The projecting of the prior identity information into the first space by the first projection unit to obtain N identity latent vectors includes: Projecting the prior identity information into the mean and variance of the first space through the first projection unit; Sampling is performed based on the mean and variance of the first space to obtain the N identity latent vectors.

4. The method according to claim 3, characterized in that The method further comprises: Determining a divergence constraint on the N identity latent vectors; Determining the loss of the target network model according to the identity anonymized image further includes: The loss of the target network model is determined based on the identity anonymized image and the divergence constraint.

5. The method according to claim 4, characterized in that Determining the loss of the target network model according to the identity anonymized image and the divergence constraint includes: The loss of the target network model is determined according to the identity anonymized image, the divergence constraint, and the identity loss.

6. The method according to claim 1, characterized in that The determining the identity loss according to the N first virtual identity vectors and the N second virtual identity vectors includes: For an i-th second virtual identity vector among the N second virtual identity vectors, use the i-th second virtual identity vector to update the virtual identity vector corresponding to the first target in the i-th dynamic list, wherein the i-th dynamic list includes virtual identity vectors of different targets at an i-th resolution, where i is a positive integer from 1 to N; Determining, according to the i-th first virtual identity vector and the updated i-th dynamic list, an identity sub-loss corresponding to the i-th first virtual identity vector; The sum of the identity sub-losses corresponding to the N first virtual identity vectors is determined as the identity loss.

7. The method according to claim 6, characterized in that The determining, according to the i-th first virtual identity vector and the updated i-th dynamic list, the identity sub-loss corresponding to the i-th first virtual identity vector includes: After comparing the i-th second virtual identity vector with a first preset value, multiplying the vector by the i-th first virtual identity vector and then performing an exponential operation to obtain a first operation value; Comparing each second virtual identity vector in the updated i-th dynamic list with the first preset value, multiplying it by the corresponding i-th first virtual identity vector and then performing an exponential operation to obtain a second operation value corresponding to each second virtual identity vector; After comparing the first operation value with the sum of the second operation values ​​corresponding to each second virtual identity vector, performing a logarithmic operation to obtain a third operation value; The negative of the third operation value is determined as the identity sub-loss corresponding to the i-th first virtual identity vector.

8. The method according to any one of claims 1 to 7, characterized in that The attribute module includes an encoding unit and a decoding unit. The second training image is processed by the attribute module in the target network model to obtain M attribute vectors, including: inputting the second training image into the encoding unit to obtain feature information of the second training image; The feature information is input into the decoding unit to obtain M attribute vectors.

9. The method according to any one of claims 1 to 7, characterized in that The fusion module includes a plurality of different resolution layers, and the step of inputting the N first virtual identity vectors and the M attribute vectors into the fusion module of the target network model to obtain an identity anonymized image of the second training image includes: According to the resolutions corresponding to the N first virtual identity vectors, the N first virtual identity vectors are used as patterns and the M attribute vectors are used as noises, which are input into the corresponding resolution layer to obtain an identity anonymized image of the second training image.

10. The method according to claim 5, characterized in that Determining the loss of the target network model according to the identity anonymized image, the divergence constraint, and the identity loss includes: determining a reconstruction loss between the identity-anonymized image and the second training image; The loss of the target network model is determined according to the reconstruction loss, the divergence constraint and the identity loss.

11. The method according to claim 10, characterized in that The method further comprises: determining a first distance between the identity-anonymized image and the first training image, a second distance between the identity-anonymized image and the second training image, and a third distance between the first training image and the second training image; determining a contrast loss based on the first distance, the second distance, and the third distance; Determining the loss of the target network model according to the reconstruction loss, the divergence constraint, and the identity loss includes: The loss of the target network model is determined according to the reconstruction loss, the divergence constraint, the identity loss, and the contrastive loss.

12. The method according to claim 11, characterized in that The determining of the contrast loss according to the first distance, the second distance, and the third distance includes: Determine a sum of a square of a difference between the second distance and the third distance and the first distance; The difference between the preset value and the sum value is determined as the contrast loss.

13. The method according to claim 11, characterized in that If the fusion module is an adversarial network, determining the loss of the target network model according to the reconstruction loss, the divergence constraint, the identity loss, and the contrast loss includes: determining an adversarial loss based on the identity anonymized image and the first training image; A weighted sum of the reconstruction loss, the divergence constraint, the identity loss, the contrast loss, and the adversarial loss is determined as the loss of the target network model.

14. A method for anonymizing identity, characterized in that: include: Sampling is performed on the target space of the projection module in the target network model to obtain N virtual identity vectors, where N is a positive integer; The image to be processed is processed by the attribute module in the target network model to obtain M attribute vectors, where M is a positive integer; Inputting the N virtual identity vectors and the M attribute vectors into a fusion module of the target network model to obtain an identity anonymized image of the image to be processed; In which, the target network model is obtained by training according to the loss of the target network model, and the loss is determined according to the identity anonymization image and identity loss of the second training image. The identity anonymization image is obtained by inputting N first virtual identity vectors and M attribute vectors of the second training image into the fusion module of the target network model. The M attribute vectors of the second training image are obtained by processing the second training image through the attribute module in the target network model. The N first virtual identity vectors are obtained by projecting the first training image into the target space through the projection module in the target network model. The identity loss is determined according to the N first virtual identity vectors and the N second virtual identity vectors. The N second virtual identity vectors are obtained by processing the third training image through the projection reference module. The third training image and the first training image are two different images of the first target. The projection reference module has the same network structure as the projection module and is updated according to the projection module.

15. The method according to claim 14, characterized in that The target space includes a first space and a second space, the target network model includes a second projection unit, and sampling is performed on the target space of the projection module in the target network model to obtain N virtual identity vectors, including: Sampling is performed on the first space to obtain N identity latent vectors; The N identity latent vectors are projected into a second space by the second projection unit to obtain the N virtual identity vectors.

16. The method according to claim 15, characterized in that The mean and variance of the first space satisfy a standard Gaussian distribution, and the sampling in the first space to obtain N identity latent vectors includes: Sampling is performed based on the mean and variance of the first space to obtain the N identity latent vectors.

17. A model training device, characterized in that: include: a projection unit, configured to project the first training image into a target space through a projection module in a target network model to obtain N first virtual identity vectors, where N is a positive integer; an attribute unit, configured to process the second training image through the attribute module in the target network model to obtain M attribute vectors, where M is a positive integer; a fusion unit, configured to input the N first virtual identity vectors and the M attribute vectors into a fusion module of the target network model to obtain an identity anonymized image of the second training image; a training unit, configured to determine a loss of the target network model based on the identity anonymized image, and train the target network model based on the loss; The projection unit is further configured to process a third training image using a projection reference module to obtain N second virtual identity vectors, wherein the third training image and the first training image are two different images of the first target, and the projection reference module has the same network structure as the projection module and is updated according to the projection module; The training unit is specifically configured to determine an identity loss based on the N first virtual identity vectors and the N second virtual identity vectors, determine a loss of the target network model based on the identity anonymized image and the identity loss, and train the target network model based on the loss.

18. An identity anonymization device, characterized in that: include: A sampling unit, configured to perform sampling on a target space of a projection module in a target network model to obtain N virtual identity vectors, where N is a positive integer; An attribute unit, configured to process the image to be processed by an attribute module in a target network model to obtain M attribute vectors, where M is a positive integer; an anonymization unit, configured to input the N virtual identity vectors and the M attribute vectors into a fusion module of the target network model to obtain an identity anonymized image of the image to be processed; In which, the target network model is obtained by training according to the loss of the target network model, and the loss is determined according to the identity anonymization image and identity loss of the second training image. The identity anonymization image is obtained by inputting N first virtual identity vectors and M attribute vectors of the second training image into the fusion module of the target network model. The M attribute vectors of the second training image are obtained by processing the second training image through the attribute module in the target network model. The N first virtual identity vectors are obtained by projecting the first training image into the target space through the projection module in the target network model. The identity loss is determined according to the N first virtual identity vectors and the N second virtual identity vectors. The N second virtual identity vectors are obtained by processing the third training image through the projection reference module. The third training image and the first training image are two different images of the first target. The projection reference module has the same network structure as the projection module and is updated according to the projection module.

19. A computing device, characterized in that including processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 13 or 14 to 16.

20. A computer-readable storage medium, characterized in that Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 13 or 14 to 16.

Images