Multimodal privacy data generation model training method, data generation method and system

Abstract

The application provides a multi-modal private data generation model training method, data generation method and system, and belongs to the technical field of data privacy protection.The application realizes the generation of RGB images and table data in a unified framework, does not need to use additional public data and an end-to-end training mode, and is connected with each other in the generation process of various types of data, thereby improving the semantic consistency of the generated data, effectively fusing the key information of different modalities to realize accurate multi-modal data discrimination;multi-angle discrimination and multi-modal matching loss are proposed, the multi-modal discrimination loss dominates the training process of the GAN model, the single-modal discrimination loss is used to assist the model in balancing the learning efficiency of different modalities, and the matching loss promotes the learning of the model on the relationship between modalities.In order to reduce the privacy leakage risk of the generation model, the differential privacy mechanism is applied to the image and table data joint generation model based on GAN, so as to balance the availability and privacy of the synthesized data.

Description

Technical Field

[0001] This invention relates to the field of data privacy protection technology, specifically to a training method, data generation method, and system for a multimodal privacy data generation model based on generative adversarial networks. Background Technology

[0002] As machine learning is increasingly applied in real-world scenarios, vast amounts of data from various fields are potentially used to train machine learning models. However, data from hospitals, governments, and other sectors contain sensitive and private information. While this data may possess significant potential analytical value, regulatory restrictions or public concerns about personal privacy prevent its direct public disclosure or sharing, hindering data analysis. To address this issue, Generative Adversarial Networks (GANs) have emerged as a novel solution that has garnered widespread attention in recent years. By generating privacy-preserving data to replace real data for publication, GANs achieve initial protection of real data. Multimodal privacy data generation aims to simultaneously generate data from multiple modalities while protecting the multimodal data used to train the generative model. As one of the most advanced generative models currently available, GANs demonstrate significant potential in data privacy protection. By simulating distributions similar to real data, GAN models can generate virtual data without a one-to-one mapping to real data, thus providing usable alternative data without revealing the original data. This alternative data, known as synthetic data, maintains data usability while reducing the risk of privacy breaches.

[0003] Despite years of research and development in the field of privacy-preserving data generation, current GAN-based research primarily focuses on unimodal privacy-preserving data generation. However, real-world data is often multimodal; for example, a patient's medical images and diagnostic records in a hospital together constitute a complete medical record. Unimodal generation models cannot be directly applied to generate these two types of data. They fail to fully capture the inherent connections between these two data types to generate semantically correct fused data, limiting their application to a wider range of real-world scenarios. Furthermore, current research on the joint generation of image and tabular data is still limited. One approach is based on a shared generator that connects table and image data, using backpropagation gradients to allow the shared generator to learn information from both images and tables simultaneously. Another approach uses a pre-trained image autoencoder to compress image features, then allows the model to learn the combined table and image features. These methods, on the one hand, do not experimentally investigate the semantic consistency between the generated images and tabular data, i.e., ensuring a one-to-one correspondence between images and corresponding tabular records in terms of information transmission; on the other hand, they do not consider implementing stricter privacy protections in the generation model. When the real data used by generative adversarial networks (GANs) contains privacy-sensitive information, generators trained directly using conventional methods may leak sensitive information from the real data under specific malicious attacks. Therefore, it is necessary to apply strict privacy protection techniques to generative models to provide stronger protection for privacy-sensitive data, and this invention also needs to be able to provide privacy protection for multiple modalities of data (images and tabular data) simultaneously.

[0004] Multimodal privacy-preserving data generation based on Generative Adversarial Networks (GANs) is a challenging cross-modal data generation task, aiming to generate multimodal data while protecting the privacy of the data used for training. Current GAN models are mainly used to generate unimodal data such as images, text, and tables, or to generate image data from text. However, research on multimodal data generation, such as the joint generation of image and table data, remains limited. While table data generated by GANs offers strong defense against re-identification attacks, current research on image and table data generation does not consider strict privacy protection. Under certain attacks, such as member inference attacks, attackers could steal sensitive real data from the published GAN generator, leading to serious privacy leaks.

[0005] In summary, existing methods for joint image and table generation based on generative adversarial networks (GANs) suffer from limited modal interaction in the generator and simplistic modal fusion techniques in the discriminator, neglecting the modal differences between image and table data. This leads to uneven learning efficiency when processing different data types, impacting generation quality. Current image and table generation methods only study the generation quality of each data type, failing to delve into generating semantically correct multimodal data. The lack of research on semantic consistency limits the model's ability to deeply understand and generate multimodal data with strong internal connections. Furthermore, existing methods do not employ stringent privacy protection techniques on multimodal GANs, posing a high risk of privacy leakage when facing malicious attacks such as member inference attacks. Summary of the Invention

[0006] The purpose of this invention is to provide a training method, data generation method and system for a multimodal privacy data generation model based on generative adversarial networks, which can efficiently jointly generate image and tabular data and improve the semantic consistency between data. It adopts a differential privacy mechanism to provide more stringent protection for model training, thereby solving at least one of the technical problems existing in the background art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for training a multimodal privacy data generation model, comprising:

[0009] Obtain paired image and table datasets;

[0010] For each record in the table, the discrete and continuous attributes are encoded and normalized respectively. The results of processing all attributes in a record are concatenated to obtain the corresponding vector representation, which together with the image forms the real data.

[0011] During the data sampling phase, in order to meet the requirements of the differential privacy sampling mechanism, non-replacement sampling is performed on the real dataset each time according to the sampling rate to obtain image-table pairs;

[0012] During the discriminator training phase, the generator parameters are frozen and synthetic data of images and tables are generated; real data and synthetic data are input into the discriminator, and the parameter gradients of each image-table pair are calculated.

[0013] In the Gaussian mechanism stage, the parameter gradient norm of each image-table pair is cropped, and noise is added to the cropped gradient. Then, the discriminator parameters are optimized through backpropagation.

[0014] During the generator training phase, the discriminator parameters are frozen, and the generator parameters are optimized using the discrimination results of the synthetic data.

[0015] Furthermore, the generator parameters are frozen, and synthetic data of images and tables are generated without calculating gradients. For synthetic data, real data, and shuffled real data, samples from the same location are taken and input into the discriminator each time to calculate multi-angle discrimination and matching loss. Backpropagation is performed on the sample-by-sample loss to calculate the parameter gradient based on each real data.

[0016] Furthermore, the discriminator gradient norm of each real sample is clipped to within a preset clipping boundary; the Gaussian noise standard deviation is calculated based on the clipping boundary, and noise is added to the parameter gradient of the real samples; the parameter gradients after applying appropriate perturbations are aggregated and the discriminator parameters are optimized.

[0017] Furthermore, with gradient calculation enabled, the generator is used to generate image and tabular data; the discriminator parameters are frozen, and the synthetic data is discriminated without calculating its gradient to obtain single-modal and multi-modal discrimination results; the overall loss of the generator is calculated and the generator parameters are optimized.

[0018] Secondly, the present invention provides a multimodal privacy data generation model training system, comprising:

[0019] The acquisition module is used to acquire paired image and tabular datasets;

[0020] The preprocessing module is used to encode and normalize the discrete and continuous attributes of each record in the table, and to concatenate the processed results of all attributes in a record to obtain the corresponding vector representation, which together with the image forms the real data.

[0021] The sampling module is used to perform non-replacement sampling on the real dataset each time according to the sampling rate to meet the requirements of the differential privacy sampling mechanism during the data sampling stage, so as to obtain image-table pairs.

[0022] The discriminator module is used during the discriminator training phase to freeze the generator parameters and generate synthetic data of images and tables; input real data and synthetic data into the discriminator to calculate the parameter gradients of each image-table pair;

[0023] The cropping module is used in the Gaussian mechanism stage to crop the parameter gradient norm of each image-table pair, add noise to the cropped gradient, and then optimize the discriminator parameters through backpropagation.

[0024] The generation module is used to freeze the discriminator parameters during the generator training phase and optimize the generator parameters using the discrimination results of synthetic data.

[0025] Terminology Explanation:

[0026] Gaussian Mixture Model: A Gaussian mixture model is a probability density model that treats data as a weighted mixture of a finite number of Gaussian distributed components, each with its own mean and covariance matrix. The model parameters are estimated by maximizing the marginal likelihood function.

[0027] Membership inference attack: Membership inference attack is a privacy attack method targeting machine learning models. It infers whether the data belongs to the model's training set by analyzing the model's output on specific data (such as predicted probability or confidence).

[0028] The beneficial effects of this invention are as follows: The proposed shared generator-based generation method enables the generation of RGB images and tabular data within a unified framework, without the need for additional public data or end-to-end training. Furthermore, the generation processes of different data types are interconnected, improving the semantic consistency of the generated data. The shared parameter weight adapter automatically assigns reasonable weight values ​​to image and tabular features to effectively fuse key information from different modalities, achieving accurate multimodal data discrimination. To bridge the modal gap between images and tables, multi-angle discrimination and multimodal matching losses are proposed. The multimodal discrimination loss dominates the training process of the GAN model, while the single-modal discrimination loss assists the model in balancing the learning efficiency of different modalities. The matching loss promotes the model's learning of intermodal relationships. To reduce the privacy leakage risk of the generative model, a differential privacy mechanism is proposed to be applied to the joint generation model of GAN-based image and tabular data, balancing the usability and privacy of the synthesized data.

[0029] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a structural diagram of the overall framework of the multimodal generator described in an embodiment of the present invention.

[0032] Figure 2 This is a structural diagram of the overall framework of the multimodal discriminator according to an embodiment of the present invention.

[0033] Figure 3 This is a flowchart illustrating the multimodal privacy data generation process based on generative adversarial networks as described in an embodiment of the present invention. Detailed Implementation

[0034] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0035] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0036] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.

[0037] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0038] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0039] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0040] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0041] This invention proposes a multimodal privacy-preserving data generation method based on generative adversarial networks (GANs), achieving efficient simultaneous generation of image and tabular data. For multimodal generation, a shared generator is designed within the GAN generator to effectively connect the generation processes of different modalities, and multimodal discrimination and matching losses are introduced into the discriminator to distinguish between real and fake data at a higher level. For privacy protection, a differential privacy framework is applied, using a Gaussian mechanism to provide strict privacy protection for the multimodal data generation of the GAN model, effectively resisting malicious behaviors such as member inference attacks.

[0042] Example 1

[0043] In this embodiment 1, a multimodal privacy data generation model training system is first provided, including: an acquisition module for acquiring paired image and table datasets; a preprocessing module for encoding and normalizing the discrete and continuous attributes of a record in each table, concatenating the processed results of all attributes in a record to obtain the corresponding vector representation, which together with the image forms real data; a sampling module for performing non-replacement sampling on the real dataset each time according to the sampling rate to meet the requirements of the differential privacy sampling mechanism during the data sampling stage, obtaining image-table pairs; a discriminator module for freezing generator parameters and generating synthetic data of images and tables during the discriminator training stage; inputting real data and synthetic data into the discriminator to calculate the parameter gradient of each image-table pair; a cropping module for cropping the parameter gradient norm of each image-table pair during the Gaussian mechanism stage, adding noise to the cropped gradient, and then optimizing the discriminator parameters through backpropagation; and a generation module for freezing discriminator parameters and optimizing generator parameters using the discrimination results of the synthetic data during the generator training stage.

[0044] In this embodiment 1, a multimodal privacy data generation model is obtained by training the model using the above-mentioned training system. The model training method includes: acquiring paired image and table datasets; encoding and normalizing the discrete and continuous attributes of a record in each table, concatenating the processed results of all attributes in a record to obtain the corresponding vector representation, which together with the image forms real data; in the data sampling stage, to meet the requirements of the differential privacy sampling mechanism, non-replacement sampling is performed on the real dataset each time according to the sampling rate to obtain image-table pairs; in the discriminator training stage, the generator parameters are frozen and synthetic data of images and tables is generated; the real data and synthetic data are input into the discriminator, and the parameter gradient of each image-table pair is calculated; in the Gaussian mechanism stage, the parameter gradient norm of each image-table pair is clipped, and noise is added to the clipped gradient, and then the discriminator parameters are optimized through backpropagation; in the generator training stage, the discriminator parameters are frozen, and the generator parameters are optimized using the discrimination results of the synthetic data.

[0045] Specifically, it includes the following steps:

[0046] Step S1: In the joint generation of RGB image and tabular data, the data includes both image and tabular data. Let... This represents a pair of images and a tabular dataset, where c, h, and w represent the image's channels, height, and width, respectively. Since RGB images are generated in this invention, therefore... .

[0047] Step S2: In the preprocessing stage, each table attribute is processed by a table data transformer to obtain a table data representation that can be input into the GAN discriminator. Then the image and the converted table Put them together to form an input set .

[0048] Step S3: In the data sampling phase, to meet the requirements of the differential privacy sampling mechanism, each time the sampling rate is determined... exist The resulting mini-batch image-table pairs, obtained through non-replacement sampling, will be used to train the GAN model.

[0049] Step S4: During the discriminator training phase, freeze the generator parameters and generate image and tabular data. ;Will and Input the discriminator and calculate the parameter gradient for each sample (image-table pair).

[0050] Step S5: In the Gaussian mechanism stage, the parameter gradient norm of each sample is pruned, noise is added to the pruned gradient, and then the discriminator parameters are optimized through backpropagation.

[0051] Step S6: During the generator training phase, freeze the discriminator parameters and use... Optimize generator parameters based on discrimination results

[0052] Repeat steps S3-S6 until the model test results meet expectations or the training iterations are reached.

[0053] Specifically, step S4 includes the following steps:

[0054] S41: Freeze generator parameters to generate image and tabular data without calculating gradients. ;

[0055] S42: Generate samples and real samples and shuffled real samples Each time, a sample from the same position is input into the discriminator to calculate the multi-angle discrimination and matching loss;

[0056] S43: Backpropagate the sample-by-sample loss to calculate the parameter gradient based on each real sample.

[0057] Specifically, step S5 includes the following steps:

[0058] S51: Prune the discriminator gradient norm of each real sample to within the preset pruning boundary;

[0059] S52: Calculate the standard deviation of Gaussian noise based on the clipping boundary, and add noise to the parameter gradient of the real sample;

[0060] S53: Aggregate the parameter gradients after applying appropriate perturbations and optimize the discriminator parameters;

[0061] Specifically, step S6 includes the following steps:

[0062] S61: With gradient calculation settings in place, use the GAN generator to generate image and tabular data. ;

[0063] S62: Freeze the discriminator parameters and discriminate the synthetic data without calculating its gradient to obtain the discrimination results for single-modality and multi-modality data;

[0064] S63: Calculate the overall loss of the generator And optimize the generator parameters.

[0065] Example 2

[0066] In this second embodiment, a novel multimodal privacy data generation model is proposed—the Multi-modal Generative Adversarial Network with PrivacyPreserving. It has four main innovations: First, it uses modality fusion technology in the discriminator to combine image and table features, using the fused features to achieve unified discrimination of multimodal data; second, the loss functions of the generator and discriminator combine the discrimination results of unimodal and multimodal modes, allowing the model to learn the distribution of multimodal data from multiple perspectives; third, it adds a pairing task as an additional task for the discriminator, meaning that in addition to judging the authenticity of the input data, the discriminator also needs to judge whether the input image and table data match, forcing the model to learn the relationships between multimodal data; fourth, it uses a differential privacy mechanism in the multimodal generative adversarial network, providing privacy protection for both the image and table data used for training, and effectively balancing the usability and privacy of the synthesized data.

[0067] Unlike the RGB image data used, which can be represented as a tensor with 3 channels, the attributes in a table may be continuous or discrete, making them unsuitable for direct input into a deep learning model for training. Appropriate methods are needed to transform each attribute before the table data can be fed into the model. In other words, for a relational table... Each record in It is necessary to use appropriate methods to convert it into numerical (or vector) representations. Therefore, this embodiment designs a table data converter to preprocess the table data. The first in Attributes The following sections explain the transformation methods applicable to model inputs based on their type.

[0068] For discrete attributes For example, attributes such as gender, occupation, and disease diagnosis results can have multiple categories of values. This invention employs one-hot encoding to encode discrete attributes. For discrete attributes... For each category, one-hot encoding assigns it a value ranging from 0 to 1. An integer between the specified length and the specified value, which will be a number of times the specified value is given. The index of the binary vector is set to 1, and the rest are set to 0. Thus, for each record... Discrete attributes Each value can be represented by a corresponding vector.

[0069] For continuous properties For example, height, weight, and blood pressure are represented as continuous values ​​in the table. Current research indicates that continuous attributes in datasets may exhibit a multimodal distribution, so a Gaussian Mixture Model (GMM) can be used to normalize the values. This method first uses GMM to cluster all values ​​of the attribute in the sample data, resulting in a cluster containing... A Gaussian distributed GMM model, where the parameters of each Gaussian distribution are expressed as follows: and For each table record Calculate the probability that it belongs to all Gaussian distributions, taking the given value. Normalization is achieved based on the Gaussian distribution with the highest probability of membership.

[0070]

[0071] Finally, a record All attributes The result after conversion Connect them to obtain the final representation of the record.

[0072]

[0073] in, Indicates the number of attributes in the table.

[0074] To establish the connection between the multimodal data generation processes, existing multimodal generation models mainly employ a shared generator. This embodiment also adopts a similar approach to implement an image and table generation model. The designed multimodal generator includes a shared generator, an image generator, and a table generator. The key to the generator in this embodiment lies in using a shared generator to connect the generation of table and image data. During model training, the shared generator can simultaneously receive information from both images and tables through backpropagation gradients, thereby learning potential feature distributions that are more conducive to generating high-quality and correlated image and table data, promoting the consistency of multimodal generated data. The designed shared generator, image generator, and table generator are described below, with the overall framework as follows: Figure 1 As shown.

[0075] Specifically, for the shared generator, consistent with typical GAN ​​model design, the generator's input is a noise vector sampled from the noise distribution. The noise is first input to the shared generator. In this process, the shared generator first performs a linear transformation on the noise through a linear layer, and then reshapes it into a small-sized feature map.

[0076]

[0077]

[0078] in, This represents the output of the linear layer in the shared generator; This represents the feature map obtained after the reshaping operation. For activation functions; This is a tensor reshaping operation.

[0079] Preliminary feature map The final output of the shared generator is obtained after three layers of transpose convolution. Batch normalization is performed after each transposed convolutional layer to share information from feature maps with different noise levels, and an activation function is used for non-linear transformation.

[0080]

[0081] in, This indicates the transpose convolution operation. . Will Converted to channel number Feature map ; and The width and height of the feature map are doubled, and the number of channels is halved. .

[0082] Specifically, for image generators, image generators It directly receives the small-sized feature map generated by the shared generator and further enlarges it. Specifically, the image generator also uses a series of transposed convolution operations, and each time the size of the output feature map is doubled, while the number of channels is increased. Similarly, batch normalization and activation functions are applied after each transposed convolutional layer.

[0083]

[0084] in, This indicates a transposed convolutional layer. , The final feature map obtained. As an intermediate result, it is used for subsequent pixel shuffle operations.

[0085] Because an improperly matched stride and kernel size in transposed convolution operations can produce checkerboard artifacts in areas with high pixel values—that is, irregular combinations of surrounding pixels—resulting in uneven brightness in the generated image. The pixel shuffle operation learns the pixel mapping function to alleviate this problem. Therefore, pixel shuffle and convolution operations are applied in later parts of the image generator to mitigate the checkerboard artifacts in the generated image.

[0086]

[0087]

[0088] The scaling factor of the Pixel Shuffle layer is set to 2. The output of this layer... The input is fed into the last convolutional layer, which, in conjunction with the pixel shuffle layer, further alleviates the checkerboard effect and reduces the number of channels to 3. The resulting feature map is then processed... The image generated by the function after activation serves as the final image generator. .

[0089] Specifically, for table generators, the output of the shared generator... The output is a small feature map, which cannot be directly input into the table generator. Therefore, the features need to be scaled down first, and the table generator uses a bottleneck layer to achieve this.

[0090] Bottleneck layer Will first Flatten it into a vector, and then project it onto the same dimension as the input noise through a linear layer.

[0091]

[0092]

[0093] in, This represents a dimensionality reduction operation on the feature map. . This represents a fully connected layer within the bottleneck layer. .

[0094] Table generator The design references the implementation of CTGAN, using a unidirectional LSTM model to generate table attributes one by one. The input to the first unit of the LSTM is the output of the bottleneck layer. The unit output is projected onto a vector representation of appropriate length according to the corresponding attribute type, and then the projection result is converted back to the attribute value through the reverse conversion method of the attribute type.

[0095]

[0096]

[0097]

[0098] in ; This represents the operation of each unit in the LSTM; This represents the attribute after conversion; .

[0099] For continuous attributes, since the Gaussian Mixture Model (GMM) is used during attribute transformation, the generator needs to generate not only the normalized numerical representation but also predict the probability that the value belongs to each Gaussian distribution. This allows it to select the Gaussian distribution with the highest probability to transform the normalized numerical representation back into the attribute's value range. For discrete attributes, the category corresponding to the index with the highest probability is selected as the attribute's value.

[0100] Current methods either determine the authenticity of multimodal inputs separately from multiple modalities, or encode images into feature vectors and treat them as a special tabular attribute. However, multimodal data contains correlation information. Compared to determining the authenticity of data from a single modality, learning the relationships between multimodal data allows the discriminator to determine the authenticity of multimodal data at a higher level.

[0101] Therefore, a multimodal discriminator is proposed in this embodiment. The input to the multimodal discriminator consists of real and generator-generated image and table data. Its overall structure adopts a dual-branch structure similar to the multimodal generator, as shown in Figure 2. Each branch first extracts features from the input image and table data and judges the authenticity of the data from a single-modal perspective. Secondly, the discriminator fuses the modal features extracted by each single-modal discriminator, and further judges the authenticity of the input data by learning the relationships between modalities. The designed image discriminator, table discriminator, and multimodal discriminator are described below.

[0102] Image discriminator Main use Each convolutional layer extracts features from the input image, and each convolutional layer resizes the feature map to its original size. The image features are then subjected to a nonlinear transformation using an activation function.

[0103]

[0104] in, ; That is, the initial input image ; These are the intermediate features extracted by the image discriminator. Based on the experimental results of DCGAN research on model training stability, the image discriminator is also designed with a fully convolutional structure and does not use any pooling operations in the model; in addition, LeakyReLU is used as the activation function to stabilize the training of the discriminator.

[0105] To obtain a discrimination result based solely on image perspective, the image discriminator sets up an output layer that uses intermediate features. The image discrimination results are obtained.

[0106]

[0107] First, the feature map is reduced in dimensionality to a vector, making it suitable as input for subsequent linear layers; finally, the linear layers are used to obtain the discrimination result for the input image. .

[0108] Table discriminator The input is a vector representation of the attributes of the tabular data after conversion. Each vector is first projected onto the same vector space through a linear layer.

[0109]

[0110] , The subsequent use of bidirectional LSTM from By extracting features from tabular data, bidirectional LSTM enables each attribute to extract useful local information from its preceding and following neighboring attributes to enhance the representation of each attribute.

[0111]

[0112] in, , Table features that integrate local information After being concatenated, the features are used as input for subsequent MLPs. The MLPs then perform multiple non-linear transformations on the tabular features to obtain intermediate features.

[0113]

[0114]

[0115] in, ; Furthermore, only on the last floor Transform the vector length, i.e. . As an intermediate feature extracted from tabular data, it is used on the one hand for The output layer is used to obtain discrimination results only for tabular data, and on the other hand, it is used for subsequent modality fusion. The former is calculated as follows:

[0116]

[0117] Multimodal discriminator This section assesses the authenticity of input data from a multimodal perspective. It first learns the fusion weights of image and table features using a multimodal weight adapter. , This module shares parameters between two types of features to learn multimodal information simultaneously and obtain more accurate weights, thereby achieving efficient utilization of effective information.

[0118]

[0119]

[0120] in, This is a weight adapter. Then, modality fusion is achieved by connecting the intermediate features obtained from the image discriminator and the table discriminator based on the fusion weights. A linear fusion layer is then used to transform the fused features, and finally, a multimodal discrimination result is obtained.

[0121]

[0122]

[0123] in, Representation layer normalization is used to capture the relationships between different features of each modality; ; The output of the last fused linear layer Used to calculate the discrimination results for multimodalities.

[0124]

[0125] In this second embodiment, a multi-angle discrimination and multi-modal matching loss is proposed. For the input real data... and synthetic data All data can be divided into image data and tabular data, denoted as... , , , For the aforementioned multimodal discriminator... Enter the actual data respectively and synthetic data The multi-angle discriminant loss function is then obtained.

[0126] (1)

[0127] (2)

[0128] (3)

[0129] Equation (1) represents the discrimination result from the perspective of the table; Equation (2) represents the discrimination result from the perspective of the image; and Equation (3) represents the discrimination result from multiple modal perspectives. Adding a single-modal discrimination loss can bridge the gap between multimodal data and balance the learning difficulty of both; the multimodal discrimination loss can improve the discriminator's learning of the relationship between modalities and dominate the overall learning process of the discriminator, promoting the consistency of generated data. In order to make the training of the multimodal GAN ​​model more stable, the model uses spectral normalization and WGAN-GP-based weight penalty during the training process. Both of these techniques require that no activation function is used on the discrimination result, but the loss value is calculated directly using the model's output.

[0130] To enable the discriminator to learn the relationships between multimodal data, this embodiment includes an additional matching task learned by the discriminator, which further utilizes the multimodal discriminator. The output of . Specifically, it is for The output is converted into a matching score using the Sigmoid function, and binary cross-entropy loss is used as the loss function. The real image data and tabular data are shuffled, and the shuffled data... Because there is no correspondence between modalities, it is related to the generated data. Negative examples are considered for matching tasks, and only real data is available. Treated as a positive example, the input data of the discriminator is denoted as The matching loss can then be calculated as follows:

[0131]

[0132] The total loss function of the discriminator is then the weighted average of the losses of the above components.

[0133]

[0134] For multimodal generators Its loss function is as follows:

[0135]

[0136]

[0137]

[0138]

[0139] Because the multimodal model has inconsistent learning capabilities for different modal data, resulting in differences in learning speed and generation performance for different data, a weighted sum of the loss function and matching loss for different modalities is used in the loss function. The learning efficiency of the model for different data is balanced by an appropriate weighted sum of the weighted sum of the loss function and matching loss for different modalities.

[0140] Generators trained using conventional methods still pose a risk of privacy leakage when faced with malicious attacks such as member inference attacks, and current work on image and tabular data generation does not consider further privacy protection for the generator. Therefore, this embodiment employs a differential privacy mechanism for privacy protection. Differential privacy mechanisms have received considerable attention in the field of deep learning due to their ability to provide strict and controllable privacy guarantees.

[0141] In the field of deep learning, a commonly used definition of differential privacy is as follows: Let the dataset be... The parameters of the deep learning model are , by all The training database consisting of subsets of is denoted as The parameter space is A deep learning training mechanism with stochastic properties takes the training dataset as input, uses gradient descent for training, and outputs the trained parameters, denoted as... This training mechanism is said to meet... If it is true for any two adjacent training sets and any parameter range The parameter distribution of its output satisfies:

[0142]

[0143] In this embodiment, the mechanism for achieving differential privacy is a Gaussian mechanism. This mechanism primarily achieves the influence of a single sample on the model by perturbing the gradient during backpropagation. Specifically, this mechanism generally first calculates the function... sensitivity That is, the range of output variation of the function for any input, which corresponds to the model in deep learning. For parameters The gradient norm. Secondly, the sensitivity of the model. Crop to the set area Then calculate the standard deviation of Gaussian noise. Finally, noise is added to the gradient, so that the influence of each sample on the model is controlled within a certain range and the parameter gradient is perturbed to a certain extent.

[0144]

[0145]

[0146] in It is a controllable noise multiplier. For , The ability to determine the level of privacy protection in differential privacy, i.e., the random algorithm under this mechanism. obey ,in .

[0147] Due to access to private data Time-difference privacy mechanisms can quantify privacy loss, and techniques such as MomentAccountant are used to record the combined privacy loss caused by each access to training data during the training process. Recent research shows that, compared to matrix accounting, RDP accounting (Renyi Differential Privacy Accoutant) provides a more compact combined privacy loss boundary, thus enabling the achievement of the same level of privacy protection. To achieve more advanced model performance. In this embodiment, the total privacy loss during the training process of the RDP accounting record multimodal GAN ​​model is also used. At the same time, the generation performance of the model was tested.

[0148] In this embodiment, a differential privacy mechanism is applied to the discriminator of the GAN. Through the post-processing characteristics of this mechanism, it is ensured that the generator also adheres to the constraints of differential privacy, thereby minimizing the risk of privacy leakage when generating image and tabular data.

[0149] In summary, the multimodal GAN ​​model training process provided in this embodiment is as follows: Figure 3 As shown, the specific steps include the following:

[0150] S1: In the joint generation of RGB image and tabular data, the data includes both image and tabular data. Let... This represents a pair of images and a tabular dataset, where c, h, and w represent the image's channels, height, and width, respectively. Since RGB images are generated in this invention, therefore... .

[0151] S2: In the preprocessing stage, each table attribute is processed by a tabular data transformer to obtain a tabular data representation that can be input into the GAN discriminator. Then the image and the converted table Put them together to form an input set .

[0152] S3: During the data sampling phase, to meet the requirements of the differential privacy sampling mechanism, each sampling is based on the sampling rate. exist The resulting mini-batch image-table pairs, obtained through non-replacement sampling, will be used to train the GAN model.

[0153] S4: During the discriminator training phase, freeze the generator parameters and generate image and tabular data. ;Will and Input the discriminator and calculate the parameter gradient for each sample (image-table pair).

[0154] S5: In the Gaussian mechanism stage, the parameter gradient norm of each sample is pruned, and noise is added to the pruned gradient. Then, the discriminator parameters are optimized through backpropagation.

[0155] S6: During the generator training phase, freeze the discriminator parameters and use... The discrimination results are used to optimize the generator parameters.

[0156] Repeat steps S3-S6 until the model test results meet expectations or the training iterations are reached.

[0157] Step S4 specifically includes the following steps:

[0158] S41: Freeze generator parameters to generate image and tabular data without calculating gradients.

[0159] S42: Generate samples and real samples and shuffled real samples Each time, samples from the same location are input into the discriminator to calculate the multi-angle discrimination and matching loss.

[0160] S43: Perform backpropagation on the sample-by-sample loss to calculate the parameter gradients based on each real sample.

[0161] Step S5 specifically includes the following steps:

[0162] S51: Prune the discriminator gradient norm of each real sample to within the preset pruning boundary.

[0163] S52: Calculate the standard deviation of Gaussian noise based on the clipping boundary, and add noise to the parameter gradient of the real sample.

[0164] S53: Aggregate the parameter gradients after applying appropriate perturbations and optimize the discriminator parameters.

[0165] Step S6 specifically includes the following steps:

[0166] S61: With gradient calculation settings in place, use the GAN generator to generate image and tabular data. ;

[0167] S62: Freeze the discriminator parameters and discriminate the synthetic data without calculating its gradient to obtain the discrimination results for single-modality and multi-modality data;

[0168] S63: Calculate the overall loss of the generator And optimize the generator parameters.

[0169] In summary, the multimodal privacy data generation method based on generative adversarial networks (GANs) described in this invention, compared to existing methods, achieves the generation of RGB images and tabular data within a unified framework, without requiring additional public data or end-to-end training. Furthermore, the generation processes of different data types are interconnected, improving the semantic consistency of the generated data. The shared-parameter weight adapter automatically assigns reasonable weight values ​​to image and tabular features to effectively fuse key information from different modalities, achieving accurate multimodal data discrimination. To bridge the modal gap between images and tables, multi-angle discrimination and multimodal matching losses are proposed. The multimodal discrimination loss dominates the training process of the GAN model, while the single-modal discrimination loss assists the model in balancing the learning efficiency of different modalities, and the matching loss promotes the model's learning of intermodal relationships. To reduce the privacy leakage risk of the generative model, a differential privacy mechanism is proposed to be applied to the joint generation model of GAN-based image and tabular data, balancing the usability and privacy of the synthesized data.

[0170] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0174] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.

Claims

1. A method for training a multimodal privacy data generation model, characterized in that, include: Obtain paired image and table datasets; For each record in the table, the discrete and continuous attributes are encoded and normalized respectively. The results of processing all attributes in a record are concatenated to obtain the corresponding vector representation, which together with the image forms the real data. During the data sampling phase, in order to meet the requirements of the differential privacy sampling mechanism, non-replacement sampling is performed on the real dataset each time according to the sampling rate to obtain image-table pairs; During the discriminator training phase, the generator parameters are frozen and synthetic data of images and tables are generated. Real data, synthetic data, and shuffled real data are input into the discriminator, and the parameter gradients for each image-table pair are calculated. Specifically, the generator parameters are frozen, and synthetic data of images and tables are generated without gradient calculation. For synthetic data, real data, and shuffled real data, samples from the same location are input into the discriminator each time, and multi-angle discrimination and matching losses are calculated. Multi-angle discrimination includes discrimination from the table perspective, discrimination from the image perspective, and discrimination from a multimodal perspective. Backpropagation is performed on the sample-by-sample loss to calculate the parameter gradients based on each image-table pair. With gradient calculation enabled, image and table data are generated using the generator. The discriminator parameters are frozen, and synthetic data is discriminated without gradient calculation to obtain single-modal and multimodal discrimination results. The overall loss of the generator is calculated, and the generator parameters are optimized. In the Gaussian mechanism stage, the parameter gradient norm of each image-table pair is cropped, and noise is added to the cropped gradient. Then, the discriminator parameters are optimized through backpropagation. During the generator training phase, the discriminator parameters are frozen, and the generator parameters are optimized using the discrimination results of the synthetic data.

2. The multimodal privacy data generation model training method according to claim 1, characterized in that, The discriminator parameter gradient norm of each image-table pair is cropped to within a preset cropping boundary; the Gaussian noise standard deviation is calculated based on the cropping boundary, and noise is added to the parameter gradient of the image-table pair; the perturbed parameter gradients are aggregated and the discriminator parameters are optimized.

3. A training system for a multimodal privacy data generation model, characterized in that, include: The acquisition module is used to acquire paired image and tabular datasets; The preprocessing module is used to encode and normalize the discrete and continuous attributes of each record in the table, and to concatenate the processed results of all attributes in a record to obtain the corresponding vector representation, which together with the image forms the real data. The sampling module is used to perform non-replacement sampling on the real dataset each time according to the sampling rate to meet the requirements of the differential privacy sampling mechanism during the data sampling stage, so as to obtain image-table pairs. The discriminator module is used during the discriminator training phase to freeze generator parameters and generate synthetic data of images and tables. Real data, synthetic data, and shuffled real data are input into the discriminator to calculate the parameter gradients for each image-table pair. Specifically, the generator parameters are frozen, and synthetic data of images and tables are generated without gradient calculation. For synthetic data, real data, and shuffled real data, samples from the same location are input into the discriminator each time to calculate multi-angle discrimination and matching losses. Multi-angle discrimination includes discrimination from the table perspective, from the image perspective, and from a multimodal perspective. Backpropagation is performed on the sample-by-sample loss to calculate the parameter gradients based on each image-table pair. With gradient calculation enabled, the generator generates image and table data. The discriminator parameters are frozen, and synthetic data is discriminated without gradient calculation to obtain single-modal and multimodal discrimination results. The overall loss of the generator is calculated, and the generator parameters are optimized. The cropping module is used in the Gaussian mechanism stage to crop the parameter gradient norm of each image-table pair, add noise to the cropped gradient, and then optimize the discriminator parameters through backpropagation. The generation module is used to freeze the discriminator parameters during the generator training phase and optimize the generator parameters using the discrimination results of synthetic data.

Images