Behavior data set privacy protection method based on generative model
Replace faces in behavioral data sets through generative models and GAN technology, which solves the problem of privacy protection affecting deep learning effects in the existing technology, and realizes the balance between data set diversity and privacy protection, which is suitable for the construction of behavioral identification data sets. .
Patent Information
- Application Number
- CN202411261297.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-12-13
AI Technical Summary
When the existing technology protects the privacy of behavioral data sets, it cannot effectively maintain the effect of deep learning, which also leads to insufficient diversity of data sets, especially in the face recognition data set. Traditional mosaic processing methods cannot meet the needs of behavioral recognition data sets. .
Using a generative model-based approach, the face is generated through the generative model and the real faces in the behavioral dataset are replaced. GAN and open source face swap technology Simswap ensure that privacy protection does not affect the learning effect of the neural network, and cross-over The entropy loss function and the sigmoid activation function are trained, adjusting the dataset to increase diversity.
It realizes effective privacy protection of behavioral data sets while ensuring the effect of deep learning, increases the diversity of data sets, and can produce different types of people's behavioral data sets, avoiding the impact of traditional methods on learning effects.
Smart Images
Figure CN119128982A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of new generation information technology, and specifically relates to a behavior data set privacy protection method based on a generative model. Background Art
[0002] In the process of making behavioral datasets, the faces in the datasets are generally real faces, which will cause unnecessary information leakage. In the processing methods of dataset privacy protection, mosaicing faces is a common method, but it is impossible to confirm whether it will affect the results of deep learning.
[0003] In the prior art, there is a Chinese invention patent application with patent number CN202210435096.9 and name "A method and electronic device for protecting the privacy of facial images", which discloses that the present application involves the field of image privacy protection technology, and provides a method and electronic device for protecting the privacy of facial images. The original facial image is formatted to obtain an original image matrix of pixels of at least one facial part that needs to be privacy processed, and a trained face generation model is used to add differential noise to the original image matrix for privacy disturbance to obtain a visual generation matrix. Since the differential noise matrix can keep the distribution of the visual generation matrix of the key parts consistent with that of the original image matrix, after replacing the corresponding key parts in the original facial image with the local facial image generated by the visual generation matrix of the key parts, it can be ensured that the obtained target facial image is a normal facial image, that is, visual goodness is maintained.
[0004] The above patent is used to protect the facial information of people in the original picture, and cannot be used to construct a behavior recognition data set. Although the image processed by the differential noise matrix is visually pleasing, this method of processing the data set will lead to a lack of diversity. Summary of the invention
[0005] In view of the above problems existing in the prior art, the present invention proposes a behavior dataset privacy protection method based on a generative model. This method performs privacy protection processing on the images in the behavior dataset while ensuring the effect of deep learning. Based on the collected behavior dataset, the present invention can perform privacy protection processing on the dataset.
[0006] In order to achieve the above effects, the technical solution adopted by the present invention is as follows:
[0007] A privacy protection method for a behavioral dataset based on a generative model includes the following steps:
[0008] S1: Collect behavior data set x0 = {x (1) ,x (2) ,...,x (m)} and traverse the pictures in the behavior recognition dataset; where x0 is the dataset picture set, x (m) is a picture in the data set, and m is the number of pictures in the data set;
[0009] S2: Determine whether the face in the image can be recognized. If not, jump to S5. If it can be recognized, execute the next step.
[0010] S3: Generate faces using a generative model;
[0011] S4: Use the faces generated by the generative model to replace the real faces in the image;
[0012] S5: Determine whether the images in the data set have been traversed. If not, jump to step S1.
[0013] S6: Finally, the behavior dataset x0'={x (1) ',x (2) ',...,x (m) '}, where x0' is the set of images in the dataset that has been processed for privacy protection, and x (m)’ is an image in the dataset after the privacy protection process is completed, and m is the number of images in the dataset.
[0014] Further, S2 comprises the following steps:
[0015] S21: Use the behavior recognition dataset for learning, set the behavior type of "taking something sideways" in the dataset to be unrecognizable for faces, set other behavior types in the dataset to be recognizable for faces, and divide the dataset into a test set and a validation set at a ratio of 10:1;
[0016] S22: Use the image recognition network to learn the data set divided in step S21, adjust the number of neurons in the output layer to 1, use sigmoid as the activation function, and adjust the loss function to the cross entropy loss function. After training, the neural network can eventually determine whether the face in the behavior data set can be recognized.
[0017] Furthermore, the formula for the activation function sigmoid is:
[0018]
[0019] Furthermore, the formula of the cross entropy loss function L is:
[0020] L=y i log(p i )+(1-y i )log(1-p i )
[0021] Where yi is a Boolean variable, which is 1 if the face of the person in the data set can be recognized, otherwise it is 0; pi is the probability of having facial features, which is the output result of the output layer of the neural network;
[0022] S23: The image recognition network in step S22 is used to judge the behavior data set that needs to be processed.
[0023] Furthermore, in step S3, a public data set is used to train a generative model, and the generative model used is GAN. The data set is used for training the GAN generative model until the GAN generative model converges, and finally the generated face is output through the GAN generative model.
[0024] Furthermore, in step S3, the convergence condition of GAN is that the discriminator D believes that the image generated by the generator G is consistent with the real image.
[0025] Furthermore, in step S4, the open source face-changing technology Simswap is used to replace the faces generated in step S3 with the faces in the behavior dataset to achieve the effect of privacy protection processing.
[0026] The advantages of the present invention are:
[0027] Compared with the traditional method of mosaicing faces in behavioral data sets, the present invention replaces the real faces in the behavioral data sets with faces generated by a generative model, which protects the privacy of the target person while not affecting the learning effect of the neural network.
[0028] This application can use this method to expand the data set, increase the diversity of the data set while protecting privacy, and produce different types of behavioral data sets based on the selected face data set.
[0029] The method of generating faces for face replacement through a generative model in this application can not only protect the privacy of the people involved in the original data set, but also expand the diversity of the data set. For example, data sets of European, Asian, African and other ethnic groups can be produced. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 The figure shows a flow chart of a behavior dataset privacy protection method based on a generative model provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0032] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0033] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0034] In the description of the present application, it should be noted that the terms "upper", "vertical", "inner", "outer", etc. indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the product of the application is usually placed when in use, or the orientation or position relationship commonly understood by those skilled in the art. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.
[0035] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "set", "install", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0036] Example 1
[0037] A privacy protection method for a behavioral dataset based on a generative model includes the following steps:
[0038] S1: Collect behavior data set x0 = {x (1) ,x (2) ,...,x (m)} and traverse the pictures in the behavior recognition dataset; where x0 is the dataset picture set, x (m) is a picture in the data set, and m is the number of pictures in the data set;
[0039] S2: Determine whether the face in the image can be recognized. If not, jump to S5. If it can be recognized, execute the next step.
[0040] S3: Generate faces using a generative model;
[0041] S4: Use the faces generated by the generative model to replace the real faces in the image;
[0042] S5: Determine whether the images in the data set have been traversed. If not, jump to step S1.
[0043] S6: Finally, the behavior dataset x0'={x (1) ',x (2) ',...,x (m) '}, where x0' is the set of images in the dataset that has been processed for privacy protection, and x (m)’ is an image in the dataset after the privacy protection process is completed, and m is the number of images in the dataset.
[0044] Further, S2 comprises the following steps:
[0045] S21: Use State Farm's Distracted Driver Dataset, a public driver behavior recognition dataset, to learn. Set the behavior type of "sideways picking up something" in the dataset as unrecognizable because there is no driver's facial information in the pictures of this behavior classification. Set other behavior types in the dataset as recognizable, and divide the dataset into a test set and a validation set at a ratio of 10:1.
[0046] S22: Use an image recognition network (such as Mobilenet, CNN, Resnet, Shufflenet, Ghostnet, etc.) to learn the data set divided in step S21, adjust the number of neurons in the output layer to 1, use sigmoid as the activation function, and adjust the loss function to the cross entropy loss function. The neural network can eventually determine whether the face in the behavior data set can be recognized through training. This step is to divide the facial information of the characters in the picture and classify the behavior data set into two categories: recognizable faces and unrecognizable faces.
[0047] Furthermore, the formula for the activation function sigmoid is:
[0048]
[0049] Furthermore, the formula of the cross entropy loss function L is:
[0050] L=y i log(p i )+(1-y i )log(1-p i )
[0051] Where yi is a Boolean variable, which is 1 if the face of the person in the data set can be recognized, otherwise it is 0; pi is the probability of having facial features, which is the output result of the output layer of the neural network;
[0052] S23: The Mobilenet image recognition network in step S22 is used to judge the behavior data set that needs to be processed.
[0053] Furthermore, in step S3, the public dataset Asian Face Age Dataset is used for generative model training, and the generative model used is GAN (Generative Adversarial Networks). The dataset is used for GAN generative model training until the GAN generative model converges, and finally the generated face is output through the GAN generative model (any face dataset and generative model can be used, such as diffusion model, GAN series, regular flow, VAE, etc.).
[0054] Furthermore, in step S3, the convergence condition of GAN is that the discriminator D believes that the image generated by the generator G is consistent with the real image.
[0055] Furthermore, in step S4, the open source face-changing technology Simswap is used to replace the faces generated in step S3 with the faces in the behavior dataset to achieve the effect of privacy protection processing.
[0056] The advantages of the present invention are:
[0057] Compared with the traditional method of mosaicing faces in behavioral data sets, the present invention replaces the real faces in the behavioral data sets with faces generated by a generative model, which protects the privacy of the target person while not affecting the learning effect of the neural network.
[0058] This application can use this method to expand the data set, increase the diversity of the data set while protecting privacy, and produce different types of behavioral data sets based on the selected face data set.
[0059] The method of generating faces for face replacement through a generative model in this application can not only protect the privacy of the people involved in the original data set, but also expand the diversity of the data set. For example, data sets of European, Asian, African and other ethnic groups can be produced.
[0060] Example 2
[0061] The embodiment of the present invention provides a behavior dataset privacy protection method based on a generative model. Figure 1 As shown, the following steps S1 to S6 are included:
[0062] S1: Collect behavior data set x0 = {x (1) ,x (2) ,...,x (m)} and traverse the images in the action recognition dataset.
[0063] S2: Determine whether the face in the image can be recognized. If not, jump to S5. If it can be recognized, execute the next step.
[0064] In the embodiment of the present invention, step S2 includes the following sub-steps S21 to S23
[0065] S21: Use State Farm's Distracted Driver Dataset, a public driver behavior recognition dataset, for learning. Set the behavior type of "sideways picking up something" in the dataset as unrecognizable because there is no driver's facial information in the pictures of this behavior classification. Set other behavior types in the dataset as recognizable. And divide the dataset into a test set and a validation set at a ratio of 10:1.
[0066] S22: Use the Mobilenet network to learn the data set divided in step S21, adjust the number of neurons in the output layer to 1, use sigmoid as the activation function, and adjust the loss function to the cross entropy loss function:
[0067] Through training, the neural network can eventually determine whether faces in behavioral data sets can be recognized.
[0068] S23: Use the Mobilenet network in step S22 to judge the behavior data set that needs to be processed.
[0069] S3: Generate faces using a generative model.
[0070] In an embodiment of the present invention, a public data set Asian Face Age Dataset is used to train a generative model. The generative model used is GAN (Generative Adversarial Networks). The data set is used for GAN training until GAN converges, and finally the generated face is output through GAN.
[0071] The convergence condition of GAN is that the condition for the convergence of the generator G and the discriminator D is that the discriminator D believes that the image generated by the generator G is consistent with the real image.
[0072] S4: Use the faces generated by the generative model to replace the real faces in the image.
[0073] In the embodiment of the present invention, the open source face-changing technology Simswap is used to replace the face generated in step S3 with the face in the behavior data set to achieve the effect of privacy protection processing.
[0074] S5: Determine whether the images in the data set have been traversed. If not, jump to S1.
[0075] S6: Finally, the behavior dataset x0'={x (1) ',x (2) ',...,x (m) '}.
[0076] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.
[0077] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0080] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
[0081] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.
Claims
1. A privacy protection method for behavioral datasets based on a generative model, characterized by: The steps include: S1: Collect behavior data set x0 = {x (1) ,x (2) ,...,x (m) } and traverse the pictures in the behavior recognition dataset; where x0 is the dataset picture set, x (m) is a picture in the data set, and m is the number of pictures in the data set; S2: Determine whether the face in the image can be recognized. If not, jump to S5. If it can be recognized, execute the next step. S3: Generate faces using a generative model; S4: Use the faces generated by the generative model to replace the real faces in the image; S5: Determine whether the images in the data set have been traversed. If not, jump to step S1; S6: Finally, the behavior dataset x0'={x (1) ',x (2) ',...,x (m) '}, where x0' is the set of images in the dataset that has been processed for privacy protection, and x (m)’ is an image in the dataset after the privacy protection process is completed, and m is the number of images in the dataset.
2. According to claim 1, a behavior dataset privacy protection method based on a generative model is characterized by: S2 includes the following steps: S21: Use the behavior recognition dataset for learning, set the behavior type of "taking something sideways" in the dataset as unrecognizable for faces, set other behavior types in the dataset as recognizable for faces, and divide the dataset into a test set and a validation set at a ratio of 10:1; S22: Use the image recognition network to learn the data set divided in step S21, adjust the number of neurons in the output layer to 1, use sigmoid as the activation function, and adjust the loss function to the cross entropy loss function. After training, the neural network can finally determine whether the face in the behavior data set can be recognized; S23: The image recognition network in step S22 is used to judge the behavior data set that needs to be processed.
3. The method for protecting privacy of behavioral datasets based on a generative model according to claim 2, characterized in that: The formula for the activation function sigmoid is:
4. The method for protecting privacy of behavioral datasets based on a generative model according to claim 2, characterized in that: The formula for the cross entropy loss function L is: L=y i log(p i )+(1-y i )log(1-p i ) Among them, yi is a Boolean variable, which is 1 if the face of the person in the data set can be recognized, otherwise it is 0; pi is the probability of having facial features, which is the output result of the output layer of the neural network.
5. The method for privacy protection of behavioral datasets based on generative models according to claim 1, characterized in that: In step S3, a public data set is used to train a generative model, and a GAN is selected as the generative model. The data set is used for training the GAN generative model until the GAN generative model converges, and finally the generated face is output through the GAN generative model.
6. The method for privacy protection of behavioral datasets based on generative models according to claim 1, characterized in that: In step S3, the convergence condition of GAN is: the discriminator D believes that the image generated by the generator G is consistent with the real image.
7. The method for privacy protection of behavioral datasets based on generative models according to claim 1, characterized in that: In step S4, the open source face-changing technology Simswap is used to replace the faces in the behavioral dataset with the faces generated in step S3 to achieve the effect of privacy protection processing.