A defense method against multi-modal inference attacks based on a generative adversarial network, an electronic device, and a storage medium

Abstract

A defense method for multi-modal inference attack based on a generative adversarial network, an electronic device and a storage medium belong to the technical field of artificial intelligence security. In order to strengthen the feature data security of the inference attack in the federal learning process, the invention sets the federal learning framework as follows: each participant declares the feature label of the local training data to the central server before each round of federal learning model training, which is used for feature alignment of federal learning; a member inference attack method is constructed, the training data of the participant is collected, then the training data of the participant is reconstructed by the attacker, the reconstructed data, the related data that can be collected and the data held by the participant itself are used as a training data set for training a multi-modal inference attack model; a multi-modal inference attack model is constructed, including a text inference attack model with text sample data and an image inference attack model with image sample data; a defense method for multi-modal inference attack is constructed for the constructed multi-modal inference attack model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, specifically relating to a defense method, electronic device and storage medium for multimodal reasoning attacks based on generative adversarial networks. Background Technology

[0002] Federated learning is a distributed machine learning framework for collaborative model training among multiple clients. It allows clients to store their sensitive data locally and share hyperparameter information only with a central server, thus protecting training data privacy.

[0003] Recent research indicates that even if clients share training information with a central server, they may still be vulnerable to hyperparameter leakage attacks, infringing on the privacy of their training data. Membership inference attacks aim to determine whether a specific training sample is present in the model's training dataset. Attackers obtain the model's parameters and gradient information, analyze its features and patterns, and design inference strategies to determine an individual's participation. Currently, membership inference attacks, both domestically and internationally, target single-modality datasets. However, in real-world federated learning scenarios, each participant's local training data may contain multiple modalities, thus limiting existing technologies. Furthermore, current attack and defense techniques primarily rely on shadow models, requiring the synthesis of training datasets using these models, which is difficult to achieve satisfactory results in complex application scenarios. Summary of the Invention

[0004] The problem this invention aims to solve is to enhance the security of characteristic data against inference attacks during federated learning. It proposes a defense method, electronic device, and storage medium for multimodal inference attacks based on generative adversarial networks.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A defense method against multimodal reasoning attacks based on generative adversarial networks includes the following steps:

[0007] S1. The basic framework of federated learning is set as follows: each participant declares the feature labels of its local training data to the central server before each round of federated learning model training, which are used for feature alignment in federated learning.

[0008] S2. Construct a member reasoning attack method, collect training data from the participants, and then the attacker reconstructs the participants' training data. The reconstructed data and the data held by the participants themselves are used as training datasets to train a multimodal reasoning attack model.

[0009] S3. Construct a multimodal reasoning attack model, including a text reasoning attack model with text as the sample data and an image reasoning attack model with images as the sample data;

[0010] S4. For the multimodal reasoning attack model constructed in step S3, construct a defense method against multimodal reasoning attacks.

[0011] Furthermore, in step S1, the i-th participant in the federated learning is set as the host. i The dataset owned by the i-th participant is X. i1 ,…,X ij , Xth ij Let x be the k-th sample in the dataset. ijk The samples are aligned according to their feature labels, and the feature positions of samples without corresponding features are set to 0. This process yields the training dataset. train .

[0012] Furthermore, the specific implementation method of step S2 includes the following steps:

[0013] S2.1. Construct a membership inference attack method. Given target data as x, target model as f, and attacker's prior knowledge as Ω, the calculation expression for the membership inference attack is:

[0014] Attacxk:{x,f,Ω}→{0,1}#(1)

[0015] Where 1 indicates that the target data x exists in the training dataset of the target model, and 0 indicates that the target data x does not exist in federated learning;

[0016] S2.2. Based on the training data of the participants collected in step S1, the attacker reconstructs the training data of the participants;

[0017] S2.3. Merge the data reconstructed in step S2.2 with the data held by the participants themselves to obtain the training dataset.

[0018] Furthermore, the specific implementation method of step S2.2 includes the following steps:

[0019] S2.2.1 Set the i-th normal participant (host) in the federated learning deep neural network model training process. i The i-th participant has n data points. i Assume there is an attacker participating in the federated learning training process, and the attacker possesses n data points. a The attacker participates in federated learning training and records parameter information for each round of the federated learning training process. i Set all normal participants nh =∑n i To build a federated learning and training process in which all normal participants and attackers participate together;

[0020] S2.2.2 Set the initial weight information w0=0 for each participant, and substitute the local data into the federated learning deep neural network model to calculate the first-round gradient information for all normal participants. Attacker's first-round gradient information The central server calculates the weight information w1 of the participants in the first round, and the calculation expression is:

[0021]

[0022] The attacker is configured to indicate that the local training data uploaded to the central server is large enough, as shown in the expression:

[0023]

[0024] Combining formulas (2) and (3), the calculation expression for the weight information of the participants in the first round is obtained as follows:

[0025]

[0026] Where η is a hyperparameter;

[0027] S2.2.3. The central server distributes w1 obtained in step S2.2.2 to all normal participants and attackers. Attackers then use the obtained w0, w1, ... The hyperparameters are calculated using the following expression:

[0028]

[0029] Based on the obtained hyperparameters, the attacker falsely uploads n... a If the value is 0, the central server calculates the weight information of the participants in the second round. The calculation expression is:

[0030]

[0031] S2.2.4. Starting from the third round, the attacker, based on step 2.2.3, obtains w2 distributed by the central server in the second round using the federated averaging algorithm, calculates η according to formula (5), and then calculates the gradient information of all normal participants in the second round. The calculation expression is:

[0032]

[0033] S2.2.5. Repeat steps S2.2.3-S2.2.4 to obtain the calculation expression for the gradient information of all normal participants in round t:

[0034]

[0035] Thus, the attacker obtains all gradient information about all normal participants from the second round to the completion of the entire federated learning training iteration.

[0036] Furthermore, the specific implementation method of step S3 includes the following steps:

[0037] S3.1. Construct a text-based reasoning attack model with sample data as text;

[0038] S3.1.1. When the sample data is text, the attacker extracts all unique words from the text to construct a vocabulary, and performs one-hot encoding on the words in the vocabulary. Each word is initialized with a zero-based vector of the same size as the vocabulary. The dimension of the corresponding vector for each word is set to 1 based on its position in the vocabulary, resulting in the i-th word vector. i for:

[0039]

[0040] S3.1.2. The attacker utilizes a deep neural network M normal The embedding layer processes field-based data, and the matrix of the embedding layer is used as model parameters, given a text dataset [word1,…,word1]. N The embedding layer only updates using words that appear in the text dataset, while the gradients of other words are 0. The expression for the output output of the deep neural network is:

[0041]

[0042]

[0043] Among them, W m×t For M normal The corresponding gradient parameter matrix, w Nt For M normal Output the value of the matrix at row N and column t, and w Nt ∈{0,1};

[0044] S3.1.3. The attacker collects the text dataset [S1,…,S] based on prior knowledge. T The text is then fed into the neural network embedding layer to calculate the output, which in turn retrieves the words that appear in the training samples of other members. The similarity between the test text and the training data of other members is measured by the size of r(output), resulting in a dataset of texts [S1,…,S] that would appear based on prior knowledge. T Similarity weight vector <w S1 ,...,w ST> The larger the weight in the similarity weight vector, the higher the probability that the corresponding text belongs to the training data;

[0045] S3.2. Construct an image-based inference attack model with image data as the sample data;

[0046] S3.2.1. When the sample data is an image, the attacker constructs an attack neural network M locally. attack M attack The neural network M participating in federated training in step S3.1 normal For samples of the same type, the number of neurons in the final fully connected output layer should be the same as the number of labels on all samples participating in federated learning.

[0047] S3.2.2. M attack As a generator in a generative adversarial neural network, it collects all images D corresponding to each label. other As part of the training data, the attacker reverse-engineers the global parameter information P based on the model parameter information sent in each round. global Combined with local training data D local Input to M attack In the expression, the calculation is as follows:

[0048]

[0049] Among them, D output To attack the computational output of the neural network, P attack This refers to the attacker's parameter information;

[0050] The discriminator calculates the distance Dis between the parameters in the attack neural network and the global parameters, expressed as:

[0051]

[0052] when At that time, D local and D other As input to the attack neural network, the corresponding confidence level α is obtained. i Sort and establish mapping tags i The calculation expression is:

[0053]

[0054] Where θ is the set confidence threshold, and satisfies θ>min(α) local This yields a training dataset D with member labels. train According to D train The binary classification convolutional neural network M is trained such that, for any given image Pic, the computational expression is:

[0055]

[0056] The output image Pic indicates whether it belongs to the training samples of the participants, where IN means it belongs and OUT means it does not.

[0057] Furthermore, the specific implementation method of step S4 includes the following steps:

[0058] S4.1. Participants randomize the original training samples to enhance the generalization ability of the local model. For image datasets, noise is added to a number of randomly selected pixels in the images. For text datasets, a number of randomly selected words in the text are replaced with other characters.

[0059] S4.2. Homomorphically encrypt the gradient information of the neural network. The following relationship exists between any two layers of the neural network, and the calculation expression is:

[0060]

[0061] in, It is the output of the m-th layer, g m It is an activation function, w i It is a weight vector, x [m-1] It is the input of the (m-1)th layer, b i It is a bias value vector;

[0062] Homomorphic encryption algorithms consist of four parts: key generation, plaintext encryption, ciphertext decryption, and homomorphic operations. The specific implementation method includes the following steps:

[0063] S4.2.1. Randomly select two prime numbers p and q of equal length that are very large, satisfying |p|=|q|=τ;

[0064] S4.2.2. Calculate N = p * q, λ = lcm(p-1, q-1), where lcm represents the least common multiple of the two numbers;

[0065] S4.2.3. Random selection Satisfy gcd(L(g) λ modN 2 (),N)=1, gcd represents the greatest common divisor of two numbers, Indicates containing N 2 A set of n integer elements, a function

[0066] S4.2.4. The public key is defined as pk = (N, g), and the private key is defined as λ;

[0067] S4.2.5. Input the public key pk and the gradient vector w of the i-th layer. Arrange all elements of the gradient vector in a row, retaining a maximum of 5 decimal places. Based on the number of decimal places for each element, add a character to the end: if there are no decimal places, add the character 'a'; if there are 1 decimal places, add the character 'b', and so on, to obtain an array of the form x1...x n 'a'(x k The structure ∈{0-9} is used to ignore the decimal point and treat it as a hexadecimal number, thus completing a one-to-one mapping of the gradient vector to integers;

[0068] S4.2.6. Assume the integer corresponding to the gradient w is m, and randomly select... Calculate the ciphertext c = g m r N (modN 2 Given the private key sk and ciphertext c, the plaintext is obtained by calculating the expression:

[0069] m=(L(c λ (modN 2 )) / L(g λ (modN 2 )t))modN#(17)

[0070] Among them, the function but The calculation is as follows:

[0071]

[0072] At this point, the attacker cannot obtain the gradient information of each layer of the global model. Introducing random bias terms into the global model will further improve security.

[0073] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a method for defending against multimodal reasoning attacks based on generative adversarial networks.

[0074] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method for defending against multimodal reasoning attacks based on generative adversarial networks.

[0075] The beneficial effects of this invention are:

[0076] This invention presents a defense method against multimodal reasoning attacks based on generative adversarial networks (GANs). This method breaks through the limitations of conventional reasoning attacks and improves upon the currently proposed single-modal reconstruction attack methods. Instead of using shadow models to synthesize fake datasets, it directly obtains the information of the real training dataset based on the neural network characteristics of federated learning. Based on this, a multimodal attack framework based on the GAN model is proposed. This novel attack framework will further enhance the research on the information of individual participants in federated learning.

[0077] The present invention describes a defense method against multimodal reasoning attacks based on generative adversarial networks. It proposes multiple defense methods against reasoning attacks, thereby effectively improving the privacy protection level of federated learning in practical application scenarios. This method has practical significance and good application prospects. Attached Figure Description

[0078] Figure 1 This is a flowchart illustrating a defense method against multimodal reasoning attacks based on generative adversarial networks, as described in this invention. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0080] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0081] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 Detailed explanation is as follows: Specific implementation method one:

[0083] A defense method against multimodal reasoning attacks based on generative adversarial networks includes the following steps:

[0084] S1. The basic framework of federated learning is set as follows: each participant declares the feature labels of its local training data to the central server before each round of federated learning model training, which are used for feature alignment in federated learning.

[0085] Furthermore, in step S1, the i-th participant in the federated learning is set as the host. i The dataset owned by the i-th participant is X. i1 ,…,X ij , Xth ij Let x be the k-th sample in the dataset. ijk The samples are aligned according to their feature labels, and the feature positions of samples without corresponding features are set to 0. This process yields the training dataset. train ;

[0086] S2. Construct a member reasoning attack method, collect training data from the participants, and then the attacker reconstructs the participants' training data. The reconstructed data and the data held by the participants themselves are used as training datasets to train a multimodal reasoning attack model.

[0087] S2.1. Construct a membership inference attack method. Given target data as x, target model as f, and attacker's prior knowledge as Ω, the calculation expression for the membership inference attack is:

[0088] Attack:{x,f,Ω}→{0,1}#(1)

[0089] Where 1 indicates that the target data x exists in the training dataset of the target model, and 0 indicates that the target data x does not exist in federated learning;

[0090] S2.2. Based on the training data of the participants collected in step S1, the attacker reconstructs the training data of the participants;

[0091] S2.3. Merge the data reconstructed in step S2.2 with the data held by the participants themselves to obtain the training dataset.

[0092] S2.2.1 Set the i-th normal participant (host) in the federated learning deep neural network model training process. i The i-th participant has n data points. i Assume there is an attacker participating in the federated learning training process, and the attacker possesses n data points. a The attacker participates in federated learning training and records parameter information for each round of the federated learning training process. i Set all normal participants n h =∑n iTo build a federated learning and training process in which all normal participants and attackers participate together;

[0093] S2.2.2 Set the initial weight information w0=0 for each participant, and substitute the local data into the federated learning deep neural network model to calculate the first-round gradient information for all normal participants. Attacker's first-round gradient information The central server calculates the weight information w1 of the participants in the first round, and the calculation expression is:

[0094]

[0095] The attacker is configured to indicate that the local training data uploaded to the central server is large enough, as shown in the expression:

[0096]

[0097] Combining formulas (2) and (3), the calculation expression for the weight information of the participants in the first round is obtained as follows:

[0098]

[0099] Where η is a hyperparameter;

[0100] S2.2.3. The central server distributes w1 obtained in step S2.2.2 to all normal participants and attackers. Attackers then use the obtained w0, w1, ... The hyperparameters are calculated using the following expression:

[0101]

[0102] Based on the obtained hyperparameters, the attacker falsely uploads n... a If the value is 0, the central server calculates the weight information of the participants in the second round. The calculation expression is:

[0103]

[0104] S2.2.4. Starting from the third round, the attacker, based on step 2.2.3, obtains w2 distributed by the central server in the second round using the federated averaging algorithm, calculates η according to formula (5), and then calculates the gradient information of all normal participants in the second round. The calculation expression is:

[0105]

[0106] S2.2.5. Repeat steps S2.2.3-S2.2.4 to obtain the calculation expression for the gradient information of all normal participants in round t:

[0107]

[0108] Thus, the attacker obtains all gradient information about all normal participants from the second round to the completion of the entire federated learning training iteration.

[0109] S3.1. Construct a text-based reasoning attack model with sample data as text;

[0110] S3.1.1. When the sample data is text, the attacker extracts all unique words from the text to construct a vocabulary, and performs one-hot encoding on the words in the vocabulary. Each word is initialized with a zero-based vector of the same size as the vocabulary. The dimension of the corresponding vector for each word is set to 1 based on its position in the vocabulary, resulting in the i-th word vector. i for:

[0111]

[0112] S3.1.2. The attacker utilizes a deep neural network M normal The embedding layer processes field-based data, and the matrix of the embedding layer is used as model parameters, given a text dataset [word1,…,word1]. N The embedding layer only updates using words that appear in the text dataset, while the gradients of other words are 0. The expression for the output output of the deep neural network is:

[0113]

[0114]

[0115] Among them, W m×t For M normal The corresponding gradient parameter matrix, w Nt For M normal Output the value of the matrix at row N and column t, and w Nt ∈{0,1};

[0116] S3.1.3. The attacker collects the text dataset [S1,…,S] based on prior knowledge. T The text is then fed into the neural network embedding layer to calculate the output, which in turn retrieves the words that appear in the training samples of other members. The similarity between the test text and the training data of other members is measured by the size of r(output), resulting in a dataset of texts [S1,…,S] that would appear based on prior knowledge. T Similarity weight vector <w S1 ,...,w ST > The larger the weight in the similarity weight vector, the higher the probability that the corresponding text belongs to the training data;

[0117] S3.2. Construct an image-based inference attack model with image data as the sample data;

[0118] S3.2.1. When the sample data is an image, the attacker constructs an attack neural network M locally. attack M attack The neural network M participating in federated training in step S3.1 normal For samples of the same type, the number of neurons in the final fully connected output layer should be the same as the number of labels on all samples participating in federated learning.

[0119] S3.2.2. M attack As a generator in a generative adversarial neural network, it collects all images D corresponding to each label. other As part of the training data, the attacker reverse-engineers the global parameter information P based on the model parameter information sent in each round. global Combined with local training data D local Input to M attack In the expression, the calculation is as follows:

[0120]

[0121] Among them, D output To attack the computational output of the neural network, P attack This refers to the attacker's parameter information;

[0122] The discriminator calculates the distance Dis between the parameters in the attack neural network and the global parameters, expressed as:

[0123]

[0124] when At that time, D local and D other As input to the attack neural network, the corresponding confidence level α is obtained. i Sort and establish mapping tags i The calculation expression is:

[0125]

[0126] Where θ is the set confidence threshold, and satisfies θ>min(α) local This yields a training dataset D with member labels. train According to D train The binary classification convolutional neural network M is trained such that, for any given image Pic, the computational expression is:

[0127]

[0128] The output image Pic indicates whether it belongs to the training samples of the participants, where IN means it belongs and OUT means it does not.

[0129] S4.1. Participants randomize the original training samples to enhance the generalization ability of the local model. For image datasets, noise is added to a number of randomly selected pixels in the images. For text datasets, a number of randomly selected words in the text are replaced with other characters.

[0130] S4.2. Homomorphically encrypt the gradient information of the neural network. The following relationship exists between any two layers of the neural network, and the calculation expression is:

[0131]

[0132] in, It is the output of the m-th layer, g m It is an activation function, w i It is a weight vector, x [m-1] It is the input of the (m-1)th layer, b i It is a bias value vector;

[0133] Homomorphic encryption algorithms consist of four parts: key generation, plaintext encryption, ciphertext decryption, and homomorphic operations. The specific implementation method includes the following steps:

[0134] S4.2.1. Randomly select two prime numbers p and q of equal length that are very large, satisfying |p|=|q|=τ;

[0135] S4.2.2. Calculate N = p * q, λ = lcm(p-1, q-1), where lcm represents the least common multiple of the two numbers;

[0136] S4.2.3. Random selection Satisfy gcd(L(g) λ modN 2 (),N)=1, gcd represents the greatest common divisor of two numbers, Indicates containing N 2 A set of n integer elements, a function

[0137] S4.2.4. The public key is defined as pk = (N, g), and the private key is defined as λ;

[0138] S4.2.5. Input the public key pk and the gradient vector w of the i-th layer. Arrange all elements of the gradient vector in a row, retaining a maximum of 5 decimal places. Based on the number of decimal places for each element, add a character to the end: if there are no decimal places, add the character 'a'; if there are 1 decimal places, add the character 'b', and so on, to obtain an array of the form x1...x n 'a'(x kThe structure ∈{0-9} is used to ignore the decimal point and treat it as a hexadecimal number, thus completing a one-to-one mapping of the gradient vector to integers;

[0139] S4.2.6. Assume the integer corresponding to the gradient w is m, and randomly select... Calculate the ciphertext c = g m r N (modN 2 Given the private key sk and ciphertext c, the plaintext is obtained by calculating the expression:

[0140] m=(L(c λ (modN 2 )) / L(g λ (modN 2 )t))modN#(17)

[0141] Among them, the function but The calculation is as follows:

[0142]

[0143] At this point, the attacker cannot obtain the gradient information of each layer of the global model. Introducing random bias terms into the global model will further improve security. Specific Implementation Method Two:

[0145] An electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of a defense method against multimodal reasoning attacks based on generative adversarial networks, as described in Specific Embodiment 1.

[0146] The computer device of the present invention may include a processor and a memory, such as a microcontroller containing a central processing unit. Furthermore, when the processor executes the computer program stored in the memory, it implements the steps of the aforementioned method for defending against multimodal reasoning attacks based on generative adversarial networks.

[0147] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0148] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices. Specific implementation method three:

[0150] A computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements a defense method against multimodal reasoning attacks based on generative adversarial networks as described in Specific Embodiment 1.

[0151] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by the processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. The computer-readable storage medium stores a computer program. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-described method for defending against multimodal reasoning attacks based on generative adversarial networks can be implemented.

[0152] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0153] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for defending against multimodal reasoning attacks based on generative adversarial networks, characterized in that, Includes the following steps: S1. The basic framework of federated learning is set as follows: each participant declares the feature labels of its local training data to the central server before each round of federated learning model training, which are used for feature alignment in federated learning. S2. Construct a member reasoning attack method, collect training data from the participants, and then the attacker reconstructs the participants' training data. The reconstructed data and the data held by the participants themselves are used as the training dataset to train the multimodal reasoning attack model. S3. Construct a multimodal reasoning attack model, including a text reasoning attack model with text as the sample data and an image reasoning attack model with images as the sample data; The specific implementation method of step S3 includes the following steps: S3.

1. Construct a text-based reasoning attack model with sample data as text; S3.1.

1. When the sample data is text, the attacker extracts all unique words from the text to construct a vocabulary, and performs one-hot encoding on the words in the vocabulary. Each word is initialized with a zero-based vector of the same size as the vocabulary. The dimension of the vector corresponding to each word is set to 1 based on its position in the vocabulary, resulting in the i-th word vector. for: ； S3.1.

2. The attacker utilizes deep neural networks The embedding layer processes field-based data, and the matrix of the embedding layer is used as a model parameter given a text dataset. The embedding layer is updated using only the words that appear in the text dataset, while the gradients of other words are zero, resulting in the output of the deep neural network. The expression is: ； ； in, for The corresponding gradient parameter matrix, for Output the value of the matrix in the Nth row and tth column, and ; S3.1.

3. The attacker collects the text dataset that will appear based on prior knowledge. And sequentially input into the neural network embedding layer for calculation. Find the words that appear in the training samples of other members, and use them to... Size measures the similarity between the test text and other members' training data, resulting in a dataset of texts that would appear based on prior knowledge. Similarity weight vector In the similarity weight vector, the larger the weight, the higher the probability that the corresponding text belongs to the training data; S3.

2. Construct an image-based inference attack model using images as sample data; S3.2.

1. When the sample data is an image, the attacker builds an attack neural network locally. , The neural networks participating in federated training in step S3.1 For samples of the same type, the number of neurons in the final fully connected output layer should be the same as the number of labels on all samples participating in federated learning. S3.2.

2. [The following appears to be a separate, unrelated sentence:] Will As a generator in a generative adversarial neural network, it collects all images corresponding to each label. As part of the training data, the attacker reverse-engineers the global parameter information based on the model parameter information sent in each round. Combined with local training data Input to In the expression, the calculation is as follows: ； in, To attack the computational output of the neural network, This refers to the attacker's parameter information; The discriminator calculates the distance between the parameters in the attack neural network and the global parameters. The calculation expression is: ； when At that time, and As input to the attack neural network, the corresponding confidence level is obtained. Sort, establish mapping The calculation expression is: ； in, It is the set confidence threshold, and satisfies This yields a training dataset with member labels. ,in accordance with Training a binary classification convolutional neural network For any given image The calculation expression is: ； Output image Whether it belongs to the training samples of the participants, among which, Indicates belonging to, This indicates that it does not belong to; S4. For the multimodal reasoning attack model constructed in step S3, construct a defense method against multimodal reasoning attacks.

2. The defense method against multimodal reasoning attacks based on generative adversarial networks according to claim 1, characterized in that, In step S1, the i-th participant in the federated learning is set as... The dataset owned by the i-th participant is , No. The k-th sample in the dataset is The samples are aligned according to their feature labels, and the feature positions of samples without corresponding features are set to 0. This process yields the training dataset. .

3. A method for defending against multimodal reasoning attacks based on generative adversarial networks according to claim 1 or 2, characterized in that, The specific implementation method of step S2 includes the following steps: S2.

1. Construct a member inference attack method, setting the given target data as... The target model is Attacker Prior knowledge is When the member inference attack is performed, the computational expression is: ； in, Represents target data It exists in the training dataset of the target model. Represents target data Does not exist In federal learning; S2.

2. Based on the training data of the participants collected in step S1, the attacker reconstructs the training data of the participants; S2.

3. Merge the data reconstructed in step S2.2 with the data held by the participants themselves to obtain the training dataset.

4. The defense method against multimodal reasoning attacks based on generative adversarial networks according to claim 3, characterized in that, The specific implementation method of step S2.2 includes the following steps: S2.2.1 Set the i-th normal participant in the federated learning deep neural network model training process. The amount of data owned by the i-th participant is Assuming there is an attacker participating in the federated learning training process. The amount of data possessed by the attacker is The attacker participates in federated learning training and records parameter information for each round of the federated learning training process. Set all normal participants To build a federated learning and training process in which all normal participants and attackers participate together; S2.2.2 Set the initial weight information of the participants The local data is then fed into the federated learning deep neural network model to calculate the first-round gradient information for all normal participants. First-round gradient information of the attacker The central server calculates the weight information of the participants in the first round. The calculation expression is: ； The attacker is configured to indicate that the local training data uploaded to the central server is large enough, as shown in the expression: ； Combining formulas (2) and (3), the calculation expression for the weight information of the participants in the first round is obtained as follows: ； in, For hyperparameters; S2.2.

3. The central server will use the data obtained in step S2.2.

2. Distributed to all normal participants and attackers, the attackers then use the information they receive... , , Calculate the hyperparameters using the following expression: ； Based on the obtained hyperparameters, the attacker falsified the upload. Then the central server calculates the weight information of the participants in the second round, and the calculation expression is: ； S2.2.

4. Starting from the third round, the attacker, based on step 2.2.3, obtains the amount distributed by the central server in the second round using the federated averaging algorithm. Calculate according to formula (5) Then, calculate the second-round gradient information for all normal participants. The calculation expression is: ； S2.2.

5. Repeat steps S2.2.3-S2.2.4 to obtain the calculation expression for the gradient information of all normal participants in round t: ； Thus, the attacker obtains all gradient information about all normal participants from the second round to the completion of the entire federated learning training iteration.

5. A method for defending against multimodal reasoning attacks based on generative adversarial networks according to claim 4, characterized in that, The specific implementation method of step S4 includes the following steps: S4.

1. Participants randomize the original training samples to enhance the generalization ability of the local model. For image datasets, noise is added to a number of randomly selected pixels in the images. For text datasets, a number of randomly selected words in the text are replaced with other characters. S4.

2. Homomorphically encrypt the gradient information of the neural network. The following relationship exists between any two layers of the neural network, and the calculation expression is: ； in, It is the first Layer output, It is an activation function. It is a weight vector. It is the first Layer input, It is a bias value vector; Homomorphic encryption algorithms consist of four parts: key generation, plaintext encryption, ciphertext decryption, and homomorphic operations. The specific implementation method includes the following steps: S4.2.

1. Randomly select two prime numbers of equal length and very large value. satisfy ; S4.2.

2. Calculation , , The least common multiple of two numbers; S4.2.

3. Random selection ,satisfy , Represents the greatest common divisor of two numbers. Indicates containing A set of n integer elements, a function ; S4.2.

4. Public key definition The private key is defined as ; S4.2.

5. Input Public Key and the Layer gradient vector Arrange all elements of the gradient vector in a row, retaining a maximum of 5 decimal places. For each element, add a character to the end based on the number of decimal places: if there are no decimal places, add the character 'a'; if there are 1 decimal places, add the character 'b', and so on, resulting in a vector like this: The structure ignores the decimal point and treats it as a hexadecimal number, completing a one-to-one mapping of the gradient vector to integers; S4.2.

6. Assuming the gradient The corresponding integer is Random selection Calculate ciphertext Enter your private key and ciphertext The plaintext is obtained, and the calculation expression is: ； Among them, the function ,but The calculation is as follows: ； At this point, the attacker cannot obtain the gradient information of each layer of the global model. Introducing random bias terms into the global model will further improve security.

6. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the defense method against multimodal reasoning attacks based on generative adversarial networks as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the defense method against multimodal reasoning attacks based on generative adversarial networks as described in any one of claims 1-5.

Images