A false information detection method and device for multimodal data privacy protection

Abstract

The present application provides a false information detection method and device with multimodal data privacy protection, which belongs to the field of information security technology, and specifically includes: a multimodal Fourier dynamic perturbation protection algorithm based on optimal neural network search, which locally encodes and fuses user multimodal data and adds Gaussian noise protection; a deep neural network model pre-training method based on a multi-head spatiotemporal correlation attention mechanism, which performs local detection training on multimodal data; a differential privacy protection mechanism based on gradient correlation enhancement, which adaptively protects the correlation of multi-layer network model gradients; a multimodal false information detection method based on a Transformer model, which detects the authenticity of user multimodal data and marks specific false areas; through the scheme disclosed in the present invention, based on the characteristics of differential privacy, the perturbed multimodal data of the present invention satisfies differential privacy, achieves high-accuracy false information detection and ensures the privacy of multimodal data.

Description

Technical Field

[0001] The present application relates to the field of information security technology, and in particular to a false information detection method and device for multimodal data privacy protection. Background Art

[0002] With the development of AI-generated content (AIGC) big models, more and more AI is able to generate false information based on real multimodal data sets, leading to serious security issues. For example, the widespread dissemination of fake news information in the media, compared with unimodal data, multimodal data (such as picture and text pairs) has a wider range of information dissemination and a greater impact on our daily lives.

[0003] Existing privacy protection technologies mainly target unimodal data privacy, without considering the privacy security of multimodal data. However, the distribution and dissemination of multimodal data in the current network are more extensive, containing more sensitive information. Protecting only unimodal data privacy cannot avoid security and privacy issues. The gradient protection method of the traditional false information detection model is mainly based on a single gradient for protection, without considering the correlation between gradients. Attackers can infer the iterative update results of the model by obtaining the correlation between gradients, thereby causing privacy leakage. In addition, the existing false information detection method based on only focuses on the authenticity of the data, but not the specific forged areas in the false data, resulting in users being unable to know the specific location area tampered by AI, and then ignoring certain potential security threats, causing more serious security problems.

[0004] Existing technologies for multimodal data privacy, such as the Transformer-based multimodal false information forgery detection method, are designed to help users detect false information generated by AI, thereby protecting users from illegal activities such as online fraud and fake news. However, this method also has the risk of user privacy leakage. Since multimodal data contains sensitive information of users, once the cloud server is attacked, this information may be leaked to the attacker. Attackers can use these multimodal data to obtain sensitive data such as personal information and browsing history, and carry out various forms of fraud such as identity theft, posing a serious threat to user security and privacy. In addition, the Transformer model based on multimodal data may also leak sensitive data of users during the detection process. Attackers can obtain sensitive information in the detection data by analyzing the correlation between model gradients, and then tamper with the detection results. Summary of the invention

[0005] The present application is made in view of the above-mentioned issues, and its purpose is to provide a false information detection method and device with multimodal data privacy protection, a multimodal Fourier dynamic perturbation protection algorithm based on optimal neural network search, which fuses user image and text data and protects the privacy of the fused multimodal data; a deep neural network model pre-training method based on a multi-head spatiotemporal correlation attention mechanism, which performs edge local model training based on multi-target loss according to user perturbation multimodal fusion embedding; a differential privacy protection mechanism based on gradient correlation enhancement, which adds Gaussian noise with adaptive gradient correlation to each layer of weight gradient in model training to protect model gradient privacy and correlation privacy; a multimodal false information detection method based on a Transformer model, which predicts the authenticity probability of each user's multimodal data. When the detection probability is greater than or equal to the detection threshold, the data is considered to be true, otherwise it is false, thereby realizing accurate false information detection services for users.

[0006] Specifically, the first aspect of the present application provides a false information detection method for multimodal data privacy protection, comprising the following steps:

[0007] Step 1: Based on the multimodal Fourier dynamic disturbance protection algorithm for optimal neural network search, two single-modal embedding features are extracted, and a multimodal fusion embedding is generated through the interaction of the two features. The fusion embedding is converted into a frequency domain representation by Fourier transform, Gaussian noise is added, and the perturbed multimodal fusion embedding is obtained by dynamic filtering and inverse Fourier transform.

[0008] Step one specifically includes: obtaining multimodal data of multiple users, encoding each image data based on VIT image coding to obtain image embedding, encoding each text data based on BERT text coding to obtain text embedding, and then extracting the basic features of the two embeddings based on the unimodal transformer, generating each user's image text multimodal embedding through a multimodal fusion layer searched by multiple optimal neural network architectures, converting the multimodal embedding into a frequency domain representation based on Fourier transform, adding Gaussian noise, filtering high-frequency irrelevant noise through dynamic filtering, and converting into time domain based on inverse Fourier transform to obtain a perturbed multimodal embedding.

[0009] Step 2: A deep neural network model pre-training method based on a multi-head spatiotemporal correlation attention mechanism is used to calculate the model's multimodal contrast loss, detection loss, image bounding box regression loss, and text sequence labeling loss based on user perturbation multimodal fusion embedding. The total model loss is aggregated to perform edge high-precision local detection training.

[0010] Step 3: Based on the differential privacy protection mechanism enhanced by gradient correlation, adaptive Gaussian noise with gradient correlation is added to the training gradient of each layer in the edge network model to achieve privacy protection based on model gradient correlation;

[0011] Step 4: A multimodal false information detection method based on the Transformer model is used to achieve high-quality false information detection on user multimodal data.

[0012] Furthermore, the step 1 specifically includes:

[0013] Step 1.1: Based on the VIT encoding model, encode the image data and convert the user's multimodal image data into an N+1-dimensional embedding vector. Based on the BERT encoding model, encode the text data and convert the user's multimodal text data into an N+1-dimensional embedding vector.

[0014] Step 1.2: According to the image and text embedding vectors, basic features are extracted based on the unimodal transformer. The two unimodal extraction features are exactly the same, and the formula is as follows:

[0015] ;

[0016] in: Basic features extracted for single-mode transformers;

[0017] for Layer image and text encoding models;

[0018] For the Basic features extracted by layer transformer;

[0019] is the pre-training parameter;

[0020] Step 1.3: According to the basic characteristics of a single modality, a multimodal fusion layer is searched through multiple optimal neural network architectures. Each multimodal fusion layer consists of an "interaction stage" and a "fusion stage". The entire multimodal fusion formula is as follows:

[0021] ;

[0022] Where: Z is the output multimodal fusion embedding;

[0023] Multimodal fusion layer for optimal neural network architecture search;

[0024] For the Layer multimodal fusion embedding;

[0025] There are two pre-training parameters;

[0026] Step 1.4: According to the multimodal fusion embedding, based on the Fourier dynamic disturbance protection method, the multimodal fusion embedding is converted into a frequency domain vector through Fourier transform, and the formula is as follows:

[0027] ;

[0028] The calculation formula of the frequency domain vector of the entire multimodal fusion embedding is as follows:

[0029] ;

[0030] in: is the i-th element after Fourier transformation in the multimodal fusion embedding;

[0031] is the nth element in the multimodal fusion embedding;

[0032] F is the frequency domain vector of multimodal fusion embedding;

[0033] N+1 is the embedding length of multimodal fusion;

[0034] j is the imaginary unit;

[0035] Step 1.5: According to the frequency domain vector embedded in the multimodal fusion, calculate the sensitivity of the current frequency domain, and add Gaussian noise to the frequency domain vector F. The formula is as follows:

[0036] ;

[0037] ;

[0038] in: is the frequency domain sensitivity;

[0039] max is to take the maximum difference;

[0040] for norm;

[0041] is the i-th element after Fourier transformation in the multimodal fusion embedding;

[0042] , is the frequency domain vector of the disturbance;

[0043] F is the frequency domain vector of multimodal fusion embedding;

[0044] , is the noise vector;

[0045] Step 1.6: According to the frequency domain vector of the disturbance, compare the frequency domain vector of the disturbance based on dynamic filtering Filter and obtain the multimodal fusion embedding of user disturbance through inverse Fourier transform. The formula is as follows:

[0046] ;

[0047] ;

[0048] ;

[0049] in: is the perturbation frequency domain vector after filtering;

[0050] K is the retention coefficient;

[0051] N+1 is the frequency domain vector length;

[0052] is the dynamic filtration ratio;

[0053] To perturb multimodal fusion embedding;

[0054] is the inverse Fourier transform.

[0055] Furthermore, the step 2 specifically includes:

[0056] Step 2.1: Perturbed multimodal fusion embedding for the user , the perturbation multimodal embedding of each user is collected at the edge layer, and the semantic consistency of the perturbation multimodal fusion embedding of different users is strengthened based on the multimodal contrastive learning method, and the contrast loss is obtained through contrastive learning training. The formula is as follows:

[0057] ;

[0058] ;

[0059] consider and The difference in direction, the formula for calculating the final multimodal contrast loss is:

[0060] ;

[0061] in: is the square of the Mahalanobis distance between the perturbed multimodal fusion embeddings of the i-th user and the j-th user;

[0062] : The mean of the perturbed multimodal fusion embedding of the i-th user;

[0063] for The covariance matrix of

[0064] for against Multimodal contrast loss;

[0065] exp() is a power function with base e;

[0066] B is the number of samples used by the edge server to train multimodal contrastive learning;

[0067] represents a hyperparameter;

[0068] for against Multimodal contrast loss;

[0069] is the multimodal contrast loss obtained for the model.

[0070] Step 2.2: For the perturbed multimodal joint embedding of the user, perform a binary classification detection operation on each multimodal embedding based on the fully connected layer and the binary classification classifier to determine whether the input is real content, and train the detection operation in the network based on the binary classification loss. The formula is as follows:

[0071] ;

[0072] ;

[0073] in: is the predicted probability of the sample;

[0074] is a binary classifier;

[0075] Embed holistic special tags for user's perturbation multimodally;

[0076] is the bias of the binary encoder;

[0077] B is the number of samples used by the edge server to train multimodal contrastive learning;

[0078] Detection loss for local forgery;

[0079] is the true label of the sample;

[0080] is the predicted probability of the sample;

[0081] Step 2.3: For the perturbed multimodal embedding of the user, if the above input is judged to be false, calculate the corresponding false area in the multimodal data, and based on the multimodal spatiotemporal self-attention mechanism, enhance the information interaction between different modalities through the spatiotemporal correlation represented by the embedding, locate the specific false areas of the picture and text through the interactive information, and calculate the total loss of the entire local false information detection model.

[0082] Furthermore, the step 2.3 specifically includes:

[0083] Step 2.31: Perturb the multimodal embedding representation according to the user , in the spatial dimension Perform mean pooling to obtain the spatial feature S. Perform word mean pooling operation to obtain word feature W, perform spatial dimension cross attention mechanism algorithm based on spatial feature S and word feature W, and select The features most correlated with S are given by the following formula:

[0084] ;

[0085] ;

[0086] ; ; ;

[0087] ;

[0088] Among them: S is the spatial feature; W is the word feature;

[0089] u is the total number of users; pool is mean pooling;

[0090] Q stands for query; K is the key; V is the value;

[0091] Embedding layer for linear feature projection;

[0092] for The spatial dimension characteristics of

[0093] () is a differentiable Select the function;

[0094] Softmax() is a normalization function;

[0095] T is the transpose operation;

[0096] is the feature dimension of query Q or key K;

[0097] Step 2.32: Perturb the multimodal embedding representation according to the user , similarly calculate the spatial feature S and word feature W, and then calculate along the time dimension Perform the first token pooling to obtain the time feature T, select b features that are most relevant to S on S, and calculate the time dimension feature. The calculation formula is as follows:

[0098] ;

[0099] ; ; ;

[0100] ;

[0101] Among them: S is the spatial feature; W is the word feature;

[0102] u is the total number of users; pool is mean pooling;

[0103] Q stands for query; K is the key; V is the value;

[0104] is the linear projection layer of word features;

[0105] It is the linear projection layer of temporal features;

[0106] () is a differentiable Select the function;

[0107] for The time dimension characteristics;

[0108] Softmax() is a normalization function;

[0109] T is the transpose operation;

[0110] is the feature dimension of query Q or key K;

[0111] Step 2.33: Perturb the multimodal embedding representation according to the user The spatial and temporal dimension features are captured based on the multi-head self-attention mechanism The spatiotemporal correlation characteristics of , and then according to The image bounding box map and text label map of each multimodal data false region are generated through a linear layer prediction, and the formula is as follows:

[0112] ;

[0113] ;

[0114] ;

[0115] in: It is the spatiotemporal correlation feature;

[0116] MultiHead() is a multi-head attention mechanism;

[0117] is a trainable embedding layer;

[0118] The bounding box map of the image predicted by the model;

[0119] Mapping of text tokens predicted by the model;

[0120] is the network layer parameter of the image;

[0121] is the network layer parameter of the text;

[0122] The network layer bias for the image;

[0123] The network layer bias for text;

[0124] sigmoid() is the network layer activation function;

[0125] Step 2.34: Calculate the predicted image bounding box based on the image bounding box map predicted by the model and the text label map The regression loss and the text sequence tag cross entropy loss are used to calculate the total loss of the entire local false information detection model based on all the loss values ​​in the above edge servers. The formula is as follows:

[0126] ;

[0127] ;

[0128] ;

[0129] in: For image border Regression loss;

[0130] Mapping of the true and false region bounding box for sample c;

[0131] is the predicted bounding box map of sample c;

[0132] B is the number of samples used by the edge server to train multimodal contrastive learning;

[0133] K is the length of the text sequence;

[0134] Labeling cross entropy loss for text sequences;

[0135] is the true tag mapping of the e-th token in sample c;

[0136] is the predicted tag mapping of the e-th token in sample c;

[0137] is the total loss of the model;

[0138] The multimodal contrast loss obtained for the model;

[0139] Loss for local forgery detection.

[0140] Furthermore, the step three specifically includes:

[0141] Step 3.1: According to the total loss of the model , the gradient of each layer of the model is calculated during the back propagation process of the model. The formula is as follows:

[0142] ;

[0143] in: is the gradient of the hth layer;

[0144] is the partial differential;

[0145] is the total loss of the model;

[0146] Step 3.2: According to the model gradient of the hth layer , based on the gradient sensitivity, the model gradient is clipped, the formula is as follows:

[0147] ;

[0148] ;

[0149] in is the clipped gradient of the hth layer;

[0150] E is sensitivity;

[0151] for norm;

[0152] is the xth element in the hth layer gradient;

[0153] Step 3.3: According to the clipped h-th layer gradient, calculate its gradient information entropy, the formula is as follows:

[0154] ;

[0155] in: Clip the gradient for layer h The gradient information entropy of

[0156] L is the total number of network layers;

[0157] For gradient The xth element in ;

[0158] Step 3.4: Based on the Mahalanobis distance between the clipped gradient of the hth layer and the clipped gradient of the h-1th layer, calculate the correlation between the gradients. The formula is as follows:

[0159] ;

[0160] in is the clipped gradient of the hth layer;

[0161] is the clipped gradient of the h-1th layer;

[0162] for and relevance;

[0163] is the covariance matrix;

[0164] Step 3.5: Based on the differential privacy protection mechanism enhanced by gradient correlation, the gradient of the hth layer is clipped Add adaptive Gaussian noise and calculate the perturbation gradient of the Fth round of local training after the training is completed. The formula is as follows:

[0165] ;

[0166] ;

[0167] ;

[0168] in: is the perturbation gradient of the Fth round of local training;

[0169] is the perturbation gradient of the hth layer calculated in the kth round;

[0170] is the perturbation gradient of the hth layer;

[0171] To satisfy Zero-mean Gaussian noise for differential privacy;

[0172] is the Gaussian noise scale of the gradient of the hth layer;

[0173] is the initial noise;

[0174] for and The maximum Mahalanobis distance between

[0175] Step 3.6: All edge servers submit the perturbation gradients to the cloud server, which aggregates all perturbation gradients for global parameter optimization and returns the optimized parameters to each edge server for edge Transformer network model update.

[0176] Furthermore, the step 4 specifically includes:

[0177] Step 4.1: According to the special mark embedded in the user's multimodal data and the global optimization parameters of the cloud server, the multimodal false information detection method based on the Transformer model is used to detect the user's multimodal false information. The formula is as follows:

[0178] ;

[0179] ;

[0180] ;

[0181] Where: G is the binary classification probability of false information detection;

[0182] sigmoid() is the sigmoid activation function;

[0183] is the activation function of the fully connected layer;

[0184] is the output of the second fully connected layer;

[0185] is the output of the first fully connected layer;

[0186] Perturb the multimodal embedding holistic features for users;

[0187] is the weight parameter of the output layer;

[0188] is the weight parameter of the second fully connected layer;

[0189] is the weight parameter of the first fully connected layer;

[0190] is the bias of the output layer;

[0191] is the bias of the second fully connected layer;

[0192] is the bias of the first fully connected layer;

[0193] is the output of the fully connected layer;

[0194] is the weight; b is the bias;

[0195] Step 4.2: Perform several rounds of iterative optimization on the edge Transformer network model and judge the data authenticity based on the detection probability of each user's multimodal data.

[0196] Furthermore, the data authenticity is judged based on the detection probability of the multimodal data of each user, specifically including that when the detection probability is greater than or equal to a detection threshold, the data is considered to be true, otherwise it is considered to be false.

[0197] A second aspect of the present application provides a false information detection device with multimodal data privacy protection, including a personal terminal device, an edge server and a cloud server;

[0198] The personal terminal device is used to implement a multi-modal Fourier dynamic disturbance protection algorithm based on optimal neural network search, and obtains multi-modal fusion embedding of disturbances through dynamic filtering and inverse Fourier transform;

[0199] The edge server is used to implement a deep neural network model pre-training method based on a multi-head spatiotemporal correlation attention mechanism, calculate the total model loss according to the multimodal fusion embedding of user perturbations to obtain the local optimal parameters of edge detection, add gradient-correlated adaptive Gaussian noise to each layer of training gradients in the edge network model based on the differential privacy protection mechanism based on gradient correlation, and finally implement high-quality false information detection on user multimodal data based on a multimodal false information detection method based on the Transformer model;

[0200] The cloud server is used to aggregate all perturbation gradients for global parameter optimization, and return the optimized parameters to each edge server for edge Transformer network model update.

[0201] In the second aspect, the present application also provides a computing device, which has the function of implementing the method described in the first aspect above. The beneficial effects can be found in the description of the first aspect, which will not be repeated here. The function can be implemented by hardware, or by executing the corresponding software implementation through hardware. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the device includes an acquisition module, a training module, and optionally, a construction module. These modules can implement the function of the training node in the method example of the first aspect above. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0202] In the third aspect, the present application also provides a computing device, which is used to implement the functions of the method described in the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The structure of the computing device includes a processor and a memory, and the memory is used to store instructions and / or data. The memory is coupled to the processor, and when the processor executes the program instructions stored in the memory, the function of the training node in the example of the first aspect above can be implemented. The structure of the computing device also includes a communication interface for communicating with other devices.

[0203] In a fourth aspect, the present application also provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0204] In a fifth aspect, the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0205] In a sixth aspect, the present application also provides a computing chip, which is connected to a memory, and the chip is used to read and execute a software program stored in the memory, and to execute the methods in the above-mentioned first aspect and various possible implementation methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0206] In order to more clearly illustrate the embodiments of the present drawings or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present drawings. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0207] Figure 1 A flowchart of the steps of a false information detection method with multimodal data privacy protection;

[0208] Figure 2 This is a structural diagram of a false information detection device with multimodal data privacy protection;

[0209] Figure 3 A comparison chart of data availability between the present application and the traditional method at different local multimodal data noise scales;

[0210] Figure 4 The figure is a comparison chart of data availability between the present application and the traditional method under different weight factors.

[0211] The purpose, features and advantages of this figure will be further described in conjunction with the embodiments with reference to the accompanying drawings. DETAILED DESCRIPTION

[0212] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0213] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0214] Unless otherwise specified, all embodiments and optional embodiments of the present application can be combined with each other to form a new technical solution.

[0215] Unless otherwise specified, all technical features and optional technical features of this application can be combined with each other to form a new technical solution.

[0216] If there is no special explanation, all steps of the present application can be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), which means that the method may include steps (a) and (b) performed sequentially, or may include steps (b) and (a) performed sequentially. For example, the method may further include step (c), which means that step (c) may be added to the method in any order. For example, the method may include steps (a), (b) and (c), or may include steps (a), (c) and (b), or may include steps (c), (a) and (b), etc.

[0217] If there is no special explanation, the "include" and "comprising" mentioned in this application are open-ended or closed-ended. For example, the "include" and "comprising" may mean that other components not listed may also be included or only the listed components may be included or only the listed components may be included.

[0218] If not specifically stated, in this application, the term "or" is inclusive. For example, the phrase "A or B" means "A, B, or both A and B". More specifically, any of the following conditions satisfies the condition "A or B": A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); or both A and B are true (or exist).

[0219] In order to better understand the solutions of the embodiments of the present application, some relevant terms and concepts that may be involved in the embodiments of the present application are first introduced below.

[0220] (1) Artificial intelligence (AI), also known as intelligent machinery or machine intelligence, refers to machines that are made by humans and can display intelligence. Generally speaking, AI refers to the technology that displays human intelligence through ordinary computer programs.

[0221] (2) Machine learning (ML). Machine learning is the core of artificial intelligence. The theory of machine learning mainly designs and analyzes some algorithms that allow computers to learn automatically. Machine learning algorithms are a type of algorithm that automatically analyzes data to obtain patterns and uses the patterns to predict unknown data. Therefore, the core of machine learning is data, algorithms (models), and computing power (computer computing power). The application areas of machine learning are very wide, including data mining, data classification, computer vision, natural language processing (NLP), biometric recognition, search engines, medical diagnosis, detection of credit card fraud, securities market analysis, DNA sequencing, speech and handwriting recognition, strategic games and robot applications. Machine learning is to design an algorithm model to process data and output the results that users want. Users can continuously tune the algorithm model to form more accurate data processing capabilities.

[0222] (3) Deep learning (DL). Deep learning is a method of machine learning. Its concept originates from the study of artificial neural networks. Multilayer perceptrons with multiple hidden layers are a type of deep learning structure, so deep learning is often referred to as deep neural networks. Compared with general machine learning, deep learning can automatically extract features, that is, automatically combine simple features into more complex features, and use these combinations to perform multi-layer weight learning to solve problems. The motivation for studying deep learning is to establish a neural network that simulates the human brain for analysis and learning. It imitates the mechanism of the human brain to interpret data, such as images, sounds, and text. Deep learning first emerged in image recognition, but in just a few years, deep learning has been extended to various fields of machine learning and has performed well. It has been applied in various fields such as image recognition, speech recognition, audio processing, natural language recognition, robot bioinformatics processing, search engines, human-computer games, online advertising targeting, medical automatic diagnosis, and finance.

[0223] (4) Neural networks (NN) are mathematical models that imitate the behavioral characteristics of animal neural networks and perform distributed parallel information processing. The purpose of information processing can be achieved by adjusting the interconnected relationships between a large number of nodes within the neural network. Neural networks have the ability to self-learn and adapt. Neural networks are usually used in artificial intelligence model training and data derivation processing.

[0224] Specifically, a neural network may usually include multiple layers connected end to end, such as convolutional layers, fully connected layers (FC), activation layers or pooling layers.

[0225] (5) Fourier transform is a mathematical tool that decomposes complex signals into sine waves or cosine waves. It is widely used in many fields, including signal processing, image analysis, physical science, and engineering computing. Its basic principle is to convert a function (or time domain signal) into a series of periodic functions for processing, thereby revealing important features of the signal, such as frequency components.

[0226] (6) The multi-head spatiotemporal correlation attention mechanism is a deep learning model that combines temporal and spatial features and is mainly used to process spatiotemporal sequence data. This mechanism can adaptively learn the important features of the data by introducing the attention mechanism and perform weighted fusion based on importance. Specifically, the multi-head attention mechanism automatically assigns attention weights to traffic sensor nodes in the road network, realizes adaptive matching of weights of different neighbor nodes, and fully obtains spatial correlation.

[0227] (7) Differential Privacy (DP) is a technology used to protect data privacy by introducing randomized noise into the data. The core idea is to ensure that the modification of any single record in the data set will not significantly affect the probability distribution of the algorithm output results, so that attackers cannot accurately infer whether a specific record exists in the data set.

[0228] In this embodiment, Figure 1 As shown, a false information detection method for multimodal data privacy protection includes the following steps:

[0229] Step 1: Based on the multimodal Fourier dynamic disturbance protection algorithm for optimal neural network search, two single-modal embedding features are extracted, and a multimodal fusion embedding is generated through the interaction of the two features. The fusion embedding is converted into a frequency domain representation by Fourier transform, Gaussian noise is added, and the perturbed multimodal fusion embedding is obtained by dynamic filtering and inverse Fourier transform.

[0230] In this embodiment, step one is specifically as follows: obtaining multimodal data of multiple users, encoding each image data based on VIT image coding to obtain image embedding, encoding each text data based on BERT text coding to obtain text embedding, and then extracting basic features of the two embeddings based on a single-mode transformer, generating image-text multimodal embedding of each user through a multimodal fusion layer searched by multiple optimal neural network architectures, converting the multimodal embedding into a frequency domain representation based on Fourier transform, adding Gaussian noise, filtering high-frequency irrelevant noise through dynamic filtering, converting into time domain based on inverse Fourier transform, and obtaining a perturbed multimodal embedding.

[0231] Step 2: A deep neural network model pre-training method based on a multi-head spatiotemporal correlation attention mechanism is used to calculate the model's multimodal contrast loss, detection loss, image bounding box regression loss, and text sequence labeling loss based on user perturbation multimodal fusion embedding. The total model loss is aggregated to perform edge high-precision local detection training.

[0232] In this embodiment, step two is specifically as follows: according to the perturbed multimodal data of each user, based on multi-objective training loss methods such as multimodal contrast loss, detection loss, image bounding box regression loss and text sequence labeling loss, the total multi-objective training loss is calculated, and local model training of the edge server is performed to obtain the local optimal parameters of edge detection.

[0233] Step 3: Based on the differential privacy protection mechanism enhanced by gradient correlation, adaptive Gaussian noise with gradient correlation is added to the training gradient of each layer in the edge network model to achieve privacy protection based on model gradient correlation;

[0234] In this embodiment, step three is specifically as follows: using the edge Transformer model to detect false information on each user's multimodal data to obtain the initial detection probability of each data, and then using the gradient features of each layer in the Transformer network model and the correlation between the current gradient and the gradient of the previous layer to calculate the perturbation gradient, and based on the perturbation gradient, locally optimize the edge Transformer model.

[0235] Step 4: A multimodal false information detection method based on the Transformer model is used to achieve high-quality false information detection on user multimodal data.

[0236] In this embodiment, step four is specifically as follows: for each user's multimodal data, all perturbed multimodal embeddings of the user, respectively, use the globally optimized Transformer network model to run the multimodal false information detection method based on the Transformer model to detect false information for the user. The multimodal false information detection method based on the Transformer model is: using the special mark in the user's perturbation multimodal embedding, the global optimization parameters of the cloud server, based on two fully connected layers and activation functions, use the Transformer network model to calculate the detection probability of each multimodal data of the user, and then iteratively optimize the edge Transformer network model based on the perturbation gradient to minimize the loss function, and obtain the optimal detection probability of each multimodal data. Finally, each optimal detection probability is judged to be greater than or equal to the detection threshold. If so, the corresponding multimodal data is considered to be true, otherwise it is false.

[0237] Furthermore, the step 1 specifically includes:

[0238] Step 1.1: Based on the VIT encoding model, encode the image data and convert the user's multimodal image data into an N+1-dimensional embedding vector. Based on the BERT encoding model, encode the text data and convert the user's multimodal text data into an N+1-dimensional embedding vector.

[0239] Step 1.2: According to the image and text embedding vectors, basic features are extracted based on the unimodal transformer. The two unimodal extraction features are exactly the same, and the formula is as follows:

[0240] ;

[0241] Step 1.3: According to the basic characteristics of a single modality, a multimodal fusion layer is searched through multiple optimal neural network architectures. Each multimodal fusion layer consists of an "interaction stage" and a "fusion stage". The entire multimodal fusion formula is as follows:

[0242] ;

[0243] In this embodiment, in step 1.3, for the multimodal fusion layer of the optimal neural network architecture search, based on dynamic The optimal neural network architecture search explores different neural network structures to find the model architecture that performs best in multimodal fusion. The specific formula is as follows:

[0244] ;

[0245] Where: GSNR is the network signal-to-noise ratio;

[0246] is the weight parameter of the network;

[0247] For the data z expectations for sampling;

[0248] Var is the value of the The variance of the sampling;

[0249] for The gradient function of

[0250] is a dynamically changing constant;

[0251] In this embodiment, for the dynamically changing constant , adaptively adjusted based on the current gradient variance and validation loss , the specific formula is as follows:

[0252] ;

[0253] ;

[0254] ;

[0255] ;

[0256] in: is a constant that is adaptively adjusted based on the current gradient variance;

[0257] is a constant that is adaptively adjusted based on the current validation loss;

[0258] is the initial constant.

[0259] Step 1.4: According to the multimodal fusion embedding, based on the Fourier dynamic disturbance protection method, the multimodal fusion embedding is converted into a frequency domain vector through Fourier transform. The formula is as follows:

[0260] ;

[0261] The calculation formula of the frequency domain vector of the entire multimodal fusion embedding is as follows:

[0262] ;

[0263] Step 1.5: According to the frequency domain vector embedded in the multimodal fusion, calculate the sensitivity of the current frequency domain, and add Gaussian noise to the frequency domain vector F. The formula is as follows:

[0264] ;

[0265] ;

[0266] Step 1.6: According to the frequency domain vector of the disturbance, compare the frequency domain vector of the disturbance based on dynamic filtering Filter and obtain the multimodal fusion embedding of user disturbance through inverse Fourier transform. The formula is as follows:

[0267] ;

[0268] ;

[0269] ;

[0270] In this embodiment, the dynamic filtering ratio is calculated based on the frequency domain energy distribution and energy change of the disturbance frequency domain vector. The specific formula is as follows:

[0271] ;

[0272] ;

[0273] ;

[0274] in: is the perturbation frequency domain vector Energy value;

[0275] is the ratio of the cumulative energy of the first k frequency components to the total energy;

[0276] is the local energy variation in the frequency domain;

[0277] Then, according to the energy distribution and energy change, the specific filtering ratio is dynamically calculated, and the formula is as follows:

[0278] ;

[0279] in: is the dynamic filtration ratio;

[0280] is the weight factor;

[0281] for Norm.

[0282] In specific applications, computer software for data processing such as Python can be used to perform the above calculations, fuse image and text data, and generate perturbed multimodal data.

[0283] It is worth mentioning that based on dynamic The optimal neural network architecture search is carried out to select the optimal multimodal fusion output to ensure the accuracy of image and text fusion, and the noise frequency domain filtering is compared based on dynamic filtering to ensure data privacy and high data availability.

[0284] Furthermore, step 2 specifically includes:

[0285] Step 2.1: Perturbed multimodal fusion embedding for the user , the perturbation multimodal embedding of each user is collected at the edge layer, and the semantic consistency of the perturbation multimodal fusion embedding of different users is strengthened based on the multimodal contrastive learning method, and the contrast loss is obtained through contrastive learning training. The formula is as follows:

[0286] ;

[0287] ;

[0288] consider and The difference in direction, the formula for calculating the final multimodal contrast loss is:

[0289] ;

[0290] Step 2.2: For the perturbed multimodal joint embedding of the user, perform a binary classification detection operation on each multimodal embedding based on the fully connected layer and the binary classification classifier to determine whether the input is real content, and train the detection operation in the network based on the binary classification loss. The formula is as follows:

[0291] ;

[0292] ;

[0293] Step 2.3: For the perturbed multimodal embedding of the user, if the above input is judged to be false, calculate the corresponding false area in the multimodal data, and based on the multimodal spatiotemporal self-attention mechanism, enhance the information interaction between different modalities through the spatiotemporal correlation represented by the embedding, locate the specific false areas of the picture and text through the interactive information, and calculate the total loss of the entire local false information detection model.

[0294] Furthermore, step 2.3 specifically includes:

[0295] Step 2.31: Perturb the multimodal joint embedding representation according to the user , in the spatial dimension Perform mean pooling to obtain the spatial feature S. Perform word mean pooling operation to obtain word feature W, perform spatial dimension cross attention mechanism algorithm based on spatial feature S and word feature W, and select The features most correlated with S are given by the following formula:

[0296] ;

[0297] ;

[0298] ; ; ;

[0299] ;

[0300] Step 2.32: Perturb the multimodal joint embedding representation according to the user , similarly calculate the spatial feature S and word feature W, and then calculate along the time dimension Perform the first token pooling to obtain the time feature T, select b features that are most relevant to S on S, and calculate the time dimension feature. The calculation formula is as follows:

[0301] ;

[0302] ; ; ;

[0303] ;

[0304] Step 2.33: Perturb the multimodal joint embedding representation according to the user The spatial and temporal dimension features are captured based on the multi-head self-attention mechanism The spatiotemporal correlation characteristics of , and then according to The image bounding box map and text label map of each multimodal data false region are generated through a linear layer prediction, and the formula is as follows:

[0305] ;

[0306] ;

[0307] ;

[0308] Step 2.34: Calculate the predicted image bounding box based on the image bounding box map predicted by the model and the text label map The regression loss and the text sequence tag cross entropy loss are used to calculate the total loss of the entire local false information detection model based on all the loss values ​​in the above edge servers. The formula is as follows:

[0309] ;

[0310] ;

[0311] ;

[0312] Furthermore, the step three specifically includes:

[0313] Step 3.1: According to the total loss of the model , the gradient of each layer of the model is calculated during the back propagation process of the model. The formula is as follows:

[0314] ;

[0315] Step 3.2: According to the model gradient of the hth layer , based on the gradient sensitivity, the model gradient is clipped, the formula is as follows:

[0316] ;

[0317] ;

[0318] Step 3.3: According to the clipped h-th layer gradient, calculate its gradient information entropy, the formula is as follows:

[0319] ;

[0320] Step 3.4: Based on the Mahalanobis distance between the clipped gradient of the hth layer and the clipped gradient of the h-1th layer, calculate the correlation between the gradients. The formula is as follows:

[0321] ;

[0322] Step 3.5: Based on the differential privacy protection mechanism enhanced by gradient correlation, the gradient of the hth layer is clipped Add adaptive Gaussian noise and calculate the perturbation gradient of the Fth round of local training after the training is completed. The formula is as follows:

[0323] ;

[0324] ;

[0325] ;

[0326] Step 3.6: All edge servers submit the perturbation gradients to the cloud server, which aggregates all perturbation gradients for global parameter optimization and returns the optimized parameters to each edge server for edge Transformer network model update.

[0327] It is worth mentioning that when simple differential privacy protection is performed on model gradients, it may lead to privacy leakage and large model accuracy loss. In contrast, gradient privacy protection based on the correlation between gradients utilizes the inherent correlation of gradients, reduces the impact on model accuracy, and enhances the privacy protection effect by optimizing the noise addition method.

[0328] In specific applications, Python and other computer software can be used to run the Transformer network model to calculate the perturbation gradient in each layer of the network.

[0329] Furthermore, step four specifically includes:

[0330] Step 4.1: According to the special mark embedded in the user's multimodal data and the global optimization parameters of the cloud server, the multimodal false information detection method based on the Transformer model is used to detect the user's multimodal false information. The formula is as follows:

[0331] ;

[0332] ;

[0333] ;

[0334] Step 4.2: Perform several rounds of iterative optimization on the edge Transformer network model and judge the data authenticity based on the detection probability of each user's multimodal data.

[0335] Furthermore, the authenticity of the data is judged based on the detection probability of the multimodal data of each user, specifically including that when the detection probability is greater than or equal to a detection threshold, the data is considered to be true, otherwise it is considered to be false.

[0336] It should be noted that the above-mentioned users are users who need to perform false information detection. Specifically, the multimodal false information detection method based on the Transformer model obtains the optimal detection probability in the user's multimodal data. When the optimal detection probability of the data is greater than or equal to the detection threshold, the data is considered to be real and not tampered with. When it is less than the detection threshold, it is considered to be forged by AI.

[0337] In order to protect the privacy of user multimodal data and model gradients, the false information detection method for multimodal data privacy protection proposed in this paper is theoretically proved to meet Local differential privacy.

[0338] Proof: Because the user perturbs the multimodal data, it satisfies the Gaussian mechanism Noise, where the privacy budget is allocated as Based on the relaxed differential privacy property, this example is based on the Fourier dynamic perturbation protection method, which satisfies Differential privacy.

[0339] Because the model perturbation gradient is added to satisfy the Gaussian mechanism Noise, where the privacy budget is allocated as Based on the relaxed differential privacy property, this example uses a gradient-correlation-enhanced differential privacy protection mechanism to satisfy Differential privacy.

[0340] Based on the serial combination principle of differential privacy, this application proposes a false information detection method for multimodal data privacy protection that meets Distributed differential privacy realizes the privacy protection of users' local multimodal social activity data and model gradients; .

[0341] In this embodiment, Figure 2 As shown, a false information detection device with multimodal data privacy protection includes a personal terminal device, an edge server and a cloud server;

[0342] The personal terminal device is used to implement a multi-modal Fourier dynamic disturbance protection algorithm based on optimal neural network search, and obtains multi-modal fusion embedding of disturbances through dynamic filtering and inverse Fourier transform;

[0343] The edge server is used to implement a deep neural network model pre-training method based on a multi-head spatiotemporal correlation attention mechanism. The total model loss is calculated based on the multimodal fusion embedding of user perturbations to obtain the local optimal parameters for edge detection. Based on the differential privacy protection mechanism enhanced by gradient correlation, adaptive Gaussian noise with gradient correlation is added to each layer of training gradients in the edge network model. Finally, a multimodal false information detection method based on the Transformer model is used to achieve high-quality false information detection on user multimodal data.

[0344] The cloud server is used to aggregate all perturbation gradients for global parameter optimization and return the optimized parameters to each edge server for edge Transformer network model update.

[0345] Specifically, when the personal terminal device participates in the false information detection service, a multimodal Fourier dynamic perturbation protection algorithm based on optimal neural network search is used to fuse the user's multimodal image text data with high precision and perturb the user's multimodal image text embedding. The privacy of user multimodal data is protected, and then the perturbed multimodal data is submitted to the edge server. The edge server uses a deep neural network model pre-training method based on a multi-head spatiotemporal correlation attention mechanism to embed user perturbation multimodality into the data. According to the multi-target spatiotemporal correlation comparison training method, local detection of the edge model is achieved. Based on the gradient correlation enhanced differential privacy protection mechanism, correlation-adaptive Gaussian noise is added to the weight gradient in model training, and the perturbation gradient is calculated to protect the model gradient and gradient correlation privacy. The perturbation gradient is then submitted to the cloud server. The cloud server aggregates all perturbation gradients for global parameter optimization and returns the optimized parameters to each edge server for model update. The edge server uses a multimodal false information detection method based on the Transformer model to detect the authenticity of each user's multimodal data, and regards the corresponding data with a detection probability greater than or equal to the detection threshold as real data, and the corresponding data with a detection probability lower than the detection threshold as forged data, thereby achieving high-quality false information detection services.

[0346] In this embodiment, Figure 3~Figure 4 As shown in the figure, based on the real multimodal news image dataset (DGM4), different parameters are used, namely the local multimodal data noise scale , weight factor , to evaluate the usability of this instance for perturbed user multimodal data.

[0347] Figure 3This is a comparison chart of data availability between this application and traditional methods at different local multimodal data noise scales. Figure 4 The data availability comparison chart of this application and the traditional method under different weight factors is shown in FIG. Figure 3 and Figure 4 The curves with circles in the middle are the data availability curves of this embodiment, and the curves with diamonds are the data availability curves of the traditional differential privacy protection method.

[0348] Depend on Figure 3 and Figure 4 It can be seen that compared with the traditional differential privacy protection method, the false information detection method for multimodal data privacy protection of the present application can well achieve the effect of improving the privacy protection of multimodal data while accurately judging the authenticity of the data, and ensure the security of multimodal data.

[0349] It should be noted that the present application is not limited to the above-mentioned embodiments. The above-mentioned embodiments are only examples, and the embodiments having the same structure as the technical idea and exerting the same effect within the scope of the technical solution of the present application are all included in the technical scope of the present application. In addition, within the scope of the subject matter of the present application, various modifications that can be thought of by those skilled in the art to the embodiments and other methods of combining some of the constituent elements in the embodiments are also included in the scope of the present application.

Claims

1. A false information detection method for multimodal data privacy protection, characterized in that: The following steps are involved: Step 1: Based on the multimodal Fourier dynamic disturbance protection algorithm for optimal neural network search, two single-modal embedding features are extracted, and a multimodal fusion embedding is generated through the interaction of the two features. The fusion embedding is converted into a frequency domain representation by Fourier transform, Gaussian noise is added, and the perturbed multimodal fusion embedding is obtained by dynamic filtering and inverse Fourier transform. Step 2: A deep neural network model pre-training method based on a multi-head spatiotemporal correlation attention mechanism is used to calculate the model's multimodal contrast loss, detection loss, image bounding box regression loss, and text sequence labeling loss based on user perturbation multimodal fusion embedding. The total model loss is aggregated to perform edge high-precision local detection training. Step 2.1: For the perturbation multimodal fusion embedding of the user, collect the perturbation multimodal embedding of each user at the edge layer, and strengthen the semantic consistency of the perturbation multimodal fusion embedding of different users based on the multimodal contrastive learning method, and obtain the contrast loss through contrastive learning training; Step 2.2: for the perturbed multimodal joint embedding of the user, perform a binary classification detection operation on each multimodal embedding based on a fully connected layer and a binary classification classifier to determine whether the input is real content, and train the detection operation in the network based on the binary classification loss; Step 2.3: For the perturbed multimodal embedding of the user, if the above input is judged to be false, calculate the corresponding false area in the multimodal data, and based on the multimodal spatiotemporal self-attention mechanism, enhance the information interaction between different modalities through the spatiotemporal correlation of the embedding representation, locate the specific false areas of the picture and text through the interactive information, and calculate the total loss of the entire local false information detection model; Step 3: Based on the differential privacy protection mechanism enhanced by gradient correlation, adaptive Gaussian noise with gradient correlation is added to the training gradient of each layer in the edge network model to achieve privacy protection based on model gradient correlation; Step 4: A multimodal false information detection method based on the Transformer model to achieve high-quality false information detection on user multimodal data; Step 4.1: According to the special mark embedded in the user's multimodal data and the global optimization parameters of the cloud server, a multimodal false information detection method based on the Transformer model is used to detect the user's multimodal false information; Step 4.2: Perform several rounds of iterative optimization on the edge Transformer network model and judge the data authenticity based on the detection probability of each user's multimodal data.

2. A false information detection method for multimodal data privacy protection according to claim 1, characterized in that: The step 1 specifically includes: Step 1.1: Based on the VIT encoding model, encode the image data to convert the user's multimodal image data into an N+1-dimensional embedding vector. Based on the BERT encoding model, encode the text data to convert the user's multimodal text data into an N+1-dimensional embedding vector. Step 1.2: According to the image and text embedding vectors, basic features are extracted based on the unimodal transformer. The two unimodal extraction features are exactly the same. The formula is as follows: ； in: Basic features extracted for single-mode transformers; for Layer image and text encoding models; For the Basic features extracted by layer transformer; is the pre-training parameter; Step 1.3: According to the basic characteristics of a single modality, a multimodal fusion layer is searched through multiple optimal neural network architectures. Each multimodal fusion layer consists of an "interaction stage" and a "fusion stage". The entire multimodal fusion formula is as follows: ； in: Multimodal fusion embedding for output; Multimodal fusion layer for optimal neural network architecture search; For the Layer multimodal fusion embedding; There are two pre-training parameters; Step 1.4: According to the multimodal fusion embedding, based on the Fourier dynamic disturbance protection method, the multimodal fusion embedding is converted into a frequency domain vector through Fourier transform, and the formula is as follows: ； The calculation formula of the frequency domain vector of the entire multimodal fusion embedding is as follows: ； in: is the i-th element after Fourier transformation in the multimodal fusion embedding; is the nth element in the multimodal fusion embedding; is the frequency domain vector embedded in multimodal fusion; +1 is the multimodal fusion embedding length; is an imaginary unit; Step 1.5: According to the frequency domain vector embedded in the multimodal fusion, calculate the sensitivity of the current frequency domain, and Add Gaussian noise, the formula is as follows: ； ； in: is the frequency domain sensitivity; To obtain the maximum difference; for norm; is the i-th element after Fourier transformation in the multimodal fusion embedding; , is the frequency domain vector of the disturbance; is the frequency domain vector embedded in multimodal fusion; , is the noise vector; Step 1.6: According to the frequency domain vector of the disturbance, compare the frequency domain vector of the disturbance based on dynamic filtering Filter and obtain the multimodal fusion embedding of user disturbance through inverse Fourier transform. The formula is as follows: ； ； ； in: is the perturbation frequency domain vector after filtering; is the retention coefficient; is the frequency domain vector length; is the dynamic filtration ratio; To perturb multimodal fusion embedding; is the inverse Fourier transform.

3. The false information detection method for multimodal data privacy protection according to claim 1, characterized in that: The step 2 specifically includes: Step 2.1: Perturbed multimodal fusion embedding for the user , the perturbation multimodal embedding of each user is collected at the edge layer, and the semantic consistency of the perturbation multimodal fusion embedding of different users is strengthened based on the multimodal contrastive learning method, and the contrast loss is obtained through contrastive learning training. The formula is as follows: ； ； consider and The difference in direction, the formula for calculating the final multimodal contrast loss is: ； in: is the square of the Mahalanobis distance between the perturbed multimodal fusion embeddings of the i-th user and the j-th user; : The mean of the perturbed multimodal fusion embedding of the i-th user; for The covariance matrix of for against Multimodal contrast loss; () is based on A power function with base ; The number of samples used to train multimodal contrastive learning for edge servers; represents a hyperparameter; for against Multimodal contrast loss; The multimodal contrast loss obtained for the model; Step 2.2: For the perturbed multimodal joint embedding of the user, perform a binary classification detection operation on each multimodal embedding based on the fully connected layer and the binary classification classifier to determine whether the input is real content, and train the detection operation in the network based on the binary classification loss. The formula is as follows: ； ； in: is the predicted probability of the sample; is a binary classifier; Embed holistic special tags for user's perturbation multimodally; is the bias of the binary encoder; The number of samples used to train multimodal contrastive learning for edge servers; Detection loss for local forgery; is the true label of the sample; Step 2.3: For the perturbed multimodal embedding of the user, if the above input is judged to be false, calculate the corresponding false area in the multimodal data, and based on the multimodal spatiotemporal self-attention mechanism, enhance the information interaction between different modalities through the spatiotemporal correlation represented by the embedding, locate the specific false areas of the picture and text through the interactive information, and calculate the total loss of the entire local false information detection model.

4. The false information detection method for multimodal data privacy protection according to claim 3 is characterized in that: The step 2.3 specifically includes: Step 2.31: Perturb the multimodal embedding representation according to the user , in the spatial dimension Perform mean pooling to obtain spatial features ,right Perform word mean pooling operation to obtain word features , according to the spatial characteristics and word features Perform spatial dimension cross attention mechanism algorithm, Select Individual and The most relevant feature is given by the following formula: ； ； ； ； ； ； in: For spatial characteristics; is the word feature; u is the total number of users; is mean pooling; Indicates a query; is the key; is the value; Embedding layer for linear feature projection; for The spatial dimension characteristics of () is a differentiable Select the function; () is the normalization function; is the transpose operation; For query or key The characteristic dimension of Step 2.32: Perturb the multimodal embedding representation according to the user , and the spatial features are also calculated and word features , and then along the time dimension Perform the first token pooling to obtain the time feature ,exist Select b and The most relevant feature is to calculate the time dimension feature. The calculation formula is as follows: ； ； ； ； ； in: For spatial characteristics; is the word feature; u is the total number of users; is mean pooling; Indicates a query; is the key; is the value; is the linear projection layer of word features; It is the linear projection layer of temporal features; () is a differentiable Select the function; for The time dimension characteristics; () is the normalization function; is the transpose operation; For query or key The characteristic dimension of Step 2.33: Perturb the multimodal embedding representation according to the user The spatial and temporal dimension features are captured based on the multi-head self-attention mechanism The spatiotemporal correlation characteristics of , and then according to The image bounding box map and text label map of each multimodal data false region are generated through a linear layer prediction, and the formula is as follows: ； ； ； in: It is the spatiotemporal correlation feature; () is the multi-head attention mechanism; is a trainable embedding layer; The bounding box map of the image predicted by the model; Mapping of text tokens predicted by the model; is the network layer parameter of the image; is the network layer parameter of the text; The network layer bias for the image; The network layer bias for text; () is the network layer activation function; Step 2.34: Calculate the predicted image bounding box based on the image bounding box map predicted by the model and the text label map The regression loss and the text sequence tag cross entropy loss are used to calculate the total loss of the entire local false information detection model based on all the loss values ​​in the above edge servers. The formula is as follows: ； ； ； in: For image border Regression loss; Mapping of the true and false region bounding box for sample c; is the predicted bounding box map of sample c; The number of samples used to train multimodal contrastive learning for edge servers; is the length of the text sequence; Labeling cross entropy loss for text sequences; is the true tag mapping of the e-th token in sample c; is the predicted tag mapping of the e-th token in sample c; is the total loss of the model; The multimodal contrast loss obtained for the model; Loss for local forgery detection.

5. The method for detecting false information with multimodal data privacy protection according to claim 1, characterized in that: The step three specifically includes: Step 3.1: According to the total loss of the model , the gradient of each layer of the model is calculated during the back propagation process of the model. The formula is as follows: ； in: is the gradient of the hth layer; is the partial differential; is the total loss of the model; Step 3.2: According to the model gradient of the hth layer , based on the gradient sensitivity, the model gradient is clipped, the formula is as follows: ； ； in is the clipped gradient of the hth layer; is sensitivity; for norm; is the xth element in the hth layer gradient; Step 3.3: According to the clipped h-th layer gradient, calculate its gradient information entropy, the formula is as follows: ； in: Clip the gradient for layer h The gradient information entropy of is the total number of network layers; For gradient The xth element in ; Step 3.4: Based on the Mahalanobis distance between the clipped gradient of the hth layer and the clipped gradient of the h-1th layer, calculate the correlation between the gradients. The formula is as follows: ； in is the clipped gradient of the hth layer; is the clipped gradient of the h-1th layer; for and relevance; is the covariance matrix; Step 3.5: Based on the differential privacy protection mechanism enhanced by gradient correlation, the gradient of the hth layer is clipped Add adaptive Gaussian noise and calculate the perturbation gradient of the Fth round of local training after the training is completed. The formula is as follows: ； ； ； in: is the perturbation gradient of the Fth round of local training; is the perturbation gradient of the hth layer calculated in the kth round; is the perturbation gradient of the hth layer; To satisfy -Zero-mean Gaussian noise for differential privacy; is the Gaussian noise scale of the gradient of the hth layer; is the initial noise; for and relevance; for and The maximum Mahalanobis distance between Step 3.6: All edge servers submit the perturbation gradients to the cloud server, which aggregates all perturbation gradients for global parameter optimization and returns the optimized parameters to each edge server for edge Transformer network model update.

6. A false information detection method for multimodal data privacy protection according to claim 1, characterized in that: The step 4 specifically includes: Step 4.1: According to the special mark embedded in the user's multimodal data and the global optimization parameters of the cloud server, the multimodal false information detection method based on the Transformer model is used to detect the user's multimodal false information. The formula is as follows: ； ； ； in: Binary classification probabilities for false information detection; () is the sigmoid activation function; is the activation function of the fully connected layer; is the output of the second fully connected layer; is the output of the first fully connected layer; Perturb the multimodal embedding holistic features for users; is the weight parameter of the output layer; is the weight parameter of the second fully connected layer; is the weight parameter of the first fully connected layer; is the bias of the output layer; is the bias of the second fully connected layer; is the bias of the first fully connected layer; Step 4.2: Perform several rounds of iterative optimization on the edge Transformer network model and judge the data authenticity based on the detection probability of each user's multimodal data.

7. A false information detection method for multimodal data privacy protection according to claim 6, characterized in that: The data authenticity is judged based on the detection probability of the multimodal data of each user, specifically including that when the detection probability is greater than or equal to a detection threshold, the data is considered to be true, otherwise it is considered to be false.

8. A false information detection device for multimodal data privacy protection, implementing a false information detection method for multimodal data privacy protection as described in any one of claims 1 to 7, characterized in that: Including personal terminal devices, edge servers and cloud servers; The personal terminal device is used to implement a multi-modal Fourier dynamic disturbance protection algorithm based on optimal neural network search, and obtains multi-modal fusion embedding of disturbances through dynamic filtering and inverse Fourier transform; The edge server is used to implement a deep neural network model pre-training method based on a multi-head spatiotemporal correlation attention mechanism, calculate the total model loss according to the multimodal fusion embedding of user perturbations to obtain the local optimal parameters of edge detection, add gradient-correlated adaptive Gaussian noise to each layer of training gradients in the edge network model based on the differential privacy protection mechanism based on gradient correlation, and finally implement high-quality false information detection on user multimodal data based on a multimodal false information detection method based on the Transformer model; The cloud server is used to aggregate all perturbation gradients for global parameter optimization, and return the optimized parameters to each edge server for edge Transformer network model update.

Images