A Community Discovery Method and Device for Multimodal Spatiotemporal Correlation Privacy Protection

Abstract

The present application provides a community discovery method and device for multi-modal spatio-temporal correlation privacy protection, which relates to the field of data security. Specifically, it includes: a multi-modal spatio-temporal contrast learning privacy protection method based on avoiding modal relaxation, which integrates multi-modal spatio-temporal feature embedding and simultaneously perturbs the multi-modal embedding by using the Laplace-Gaussian mechanism; a multi-level multi-modal user association enhancement method to obtain multi-level spatio-temporal correlations within and between modalities, enhancing the multi-community user interaction features; an adaptive decay-based gradient spatio-temporal correlation protection method to protect the spatio-temporal correlation of the model gradient and ensure the robustness of model training; and spatio-temporal community discovery based on graph neural networks to achieve highly available multi-modal community discovery. Through the solution of the present disclosure, based on the property of differential privacy, the method of the present invention satisfies differential privacy and improves the security and effectiveness of multi-modal community discovery.

Description

Technical Field

[0001] The present invention relates to the fields of information security and social network technology, and particularly relates to a community discovery method and device for multi-modal spatio-temporal correlation privacy protection. Background Art

[0002] At present, with the rapid development of artificial intelligence and big data technologies, community discovery, as an important tool for social network analysis and artificial intelligence construction, significantly improves the efficiency of social governance and resource optimization through collecting multi-source and multi-modal data and performing complex association analysis, pattern recognition, and intelligent decision-making. Community discovery can accurately identify and analyze community structures and support urban managers in making scientific decisions. For example, using various data sources such as social media, geographical location information, and traffic flow data, community discovery algorithms can reveal the community relationships and dynamic change trends in different regions of a city. However, with the increase in data volume and data diversity, the security of community discovery faces many security risk challenges.

[0003] During the community discovery process, a large amount of multi-modal data of users is involved, including social interaction records, geographical location information, activity trajectories, etc. These data not only contain rich community structure information but also contain the multi-modal spatio-temporal characteristics of community users, and thus may expose the sensitive privacy of users. For example, the geographical location of a user and their social activities often show strong spatio-temporal correlation, and an attacker can infer the user's personal sensitive privacy (such as personal travel trajectories, frequently visited locations, social circles, etc.) by analyzing the time and space correlations therein. Therefore, how to ensure the spatio-temporal correlation privacy of multi-modal data while performing community discovery has become a key problem that urgently needs to be solved.

[0004] Existing privacy protection technologies mainly include anonymization methods, homomorphic encryption methods, and traditional differential privacy methods. However, for large-scale multi-modal spatio-temporal data, anonymization methods are often difficult to resist complex data spatio-temporal correlation attacks and cannot effectively protect user privacy. The homomorphic encryption method has problems such as high computational overhead and long response time, and is limited to users with keys to access data, restricting the wide application of data. Traditional differential privacy methods mainly target single-modal data and are difficult to handle community discovery tasks with multi-modal spatio-temporal correlation characteristics. If noise is directly added to the spatio-temporal correlation of multi-modal data in community discovery, the data usability will be significantly reduced. In addition, these methods usually provide a unified privacy protection level for all users, ignoring the complexity of multi-modal data and the personalized privacy needs of users, resulting in difficulty in balancing privacy protection and data availability in practical applications. Summary of the Invention

[0005] The present invention is made in view of the above problems, and its object is to provide a community discovery method and device for multi-modal spatio-temporal correlation privacy protection, based on a multi-level multi-modal user association enhancement method to obtain multi-level spatio-temporal correlations within and between modalities, enhancing the multi-community user interaction characteristics; based on an adaptive decay gradient spatio-temporal correlation protection method to protect the spatio-temporal correlation of model gradients and ensure the robustness of model training; based on spatio-temporal community discovery using graph neural networks to achieve usable multi-modal community discovery, and based on the property of differential privacy, satisfying differential privacy, and improving the security and effectiveness of multi-modal community discovery.

[0006] Specifically, the first aspect of the present invention provides a community discovery method for multi-modal spatio-temporal correlation privacy protection, including the following steps:

[0007] Step 1: Based on a multi-modal spatio-temporal contrast learning privacy protection method that avoids modality relaxation, extract the spatio-temporal correlations of the multi-modal data of local community users, avoid the interference of modality relaxation and fuse multi-modal embeddings, and add Laplace-Gaussian noise to protect the privacy of the spatio-temporal correlations of the multi-modal fusion embeddings;

[0008] Multi-modal data refers to a data set containing various types of information, such as images, texts, audio, etc.

[0009] Laplace-Gaussian noise is a mixed noise model that combines the characteristics of the Laplace distribution and the Gaussian distribution. The Laplace distribution has a sharp peak, while the Gaussian distribution is relatively smooth. This noise model is often used to represent complex noise environments, especially in image processing and signal processing, and is widely used in differential privacy because it can provide stronger privacy protection.

[0010] Step 2: Based on a multi-level multi-modal user association enhancement method, construct a spatio-temporal association enhancement matrix based on the multi-modal fusion embeddings of multi-community users, thereby enhancing the representation of user interaction characteristics;

[0011] Step 3: Based on an adaptive decay gradient spatio-temporal correlation protection method, introduce adaptive decay Gaussian noise to protect the privacy of spatio-temporal gradients;

[0012] Step 4: Based on a spatio-temporal community discovery method using graph neural networks, deeply explore the spatio-temporal interaction characteristics between users and communities, and between users and users, and conduct multi-modal community discovery.

[0013] Furthermore, the multimodal spatio-temporal contrastive learning privacy protection method based on avoiding modal relaxation is specifically to alternately learn the unimodal information of local community users through random probability, extract the spatio-temporal correlation between modalities using a deep graph information encoder, and fuse multimodal feature embeddings according to the modal information and spatio-temporal correlation features.

[0014] Furthermore, Step 1 includes the following steps:

[0015] Step 1.1: Given the personal text modal data contained in the user and the numerical weight data for interacting with other community users, randomly select any modality for alternating learning according to random probability;

[0016] Step 1.2: Use a deep graph information encoder to extract the spatio-temporal correlation between modalities, and optimize the learning result by minimizing the prediction risk function of this modality. The formula is as follows:

[0017] ;

[0018] ;

[0019] Where: is the spatio-temporal extraction function;

[0020] is the deep graph information encoder;

[0021] is to randomly select any modality for alternating learning according to random probability ;

[0022] is the prediction risk function;

[0023] is the expectation;

[0024] is the calculation of the loss function;

[0025] is the modal data;

[0026] are the learnable parameters in the Transformer encoder;

[0027] are the learnable parameters in the information encoder;

[0028] is the true label;

[0029] During the extraction and learning process of spatio-temporal correlation, gradient alignment is used to ensure that the gradient update direction is orthogonal to the prior modal feature direction, avoiding the problem of modal relaxation in model learning. The formula is as follows:

[0030] ;

[0031] ;

[0032] ;

[0033] Wherein: is the gradient of the th network layer;

[0034] is the gradient of the th network layer;

[0035] is the learning rate;

[0036] is the gradient operator;

[0037] is the prediction risk function;

[0038] is the selection weight of the th round;

[0039] is the selection weight of the th round;

[0040] is the update factor;

[0041] is to alternately learn any modality according to the random probability ;

[0042] is the transpose operation;

[0043] is the smoothing parameter;

[0044] Step 1.3: Based on the learned single-modal information and the spatio-temporal correlation between modalities, fuse the multi-modal feature information through weighted contrast learning. The formula is as follows:

[0045] ;

[0046] ;

[0047] Wherein: is the value of the multi-modal fusion embedding;

[0048] is the spatio-temporal contribution weight;

[0049] is a spatio-temporal extraction function;

[0050] is modal data;

[0051] are learnable parameters in the Transformer encoder;

[0052] are learnable parameters in the information encoder;

[0053] is an exponential function;

[0054] is the cosine similarity between two embedding vectors;

[0055] is the modal temporal feature obtained by the Transformer encoder;

[0056] is the modal spatial feature obtained by the Transformer encoder;

[0057] is a temperature parameter;

[0058] is the batch size;

[0059] is the k-th sample in the batch;

[0060] is the k-th data value in;

[0061] Finally, the multi-modal spatio-temporal fusion embedding is obtained through weighted multi-modal fusion , is the data dimension;

[0062] Step 1.4: According to the obtained multi-modal spatio-temporal fusion embedding, the spatio-temporal correlation privacy of the fusion embedding is protected through the Laplace-Gaussian mechanism. First, calculate the importance parameter of the user in the social network community according to the number of other community users interacted by the user, and dynamically allocate the privacy budget of the user according to the importance parameter. The formula is as follows:

[0063] ;

[0064] ;

[0065] Among them: is the user 's importance parameter in the social network community;

[0066] The number of other community users currently interacting with the user; The minimum number of other community users the user has interacted with historically;

[0067] The number of other community users currently interacting with the user; The minimum number of other community users the user has interacted with historically;

[0068] The number of other community users currently interacting with the user; The maximum number of other community users the user has interacted with historically;

[0069] The number of other community users currently interacting with the user; The privacy budget allocated to the user;

[0070] The total privacy budget;

[0071] The total number of all community users;

[0072] Then, perturb the multi-modal spatio-temporal fusion embedding of the user based on the Laplace-Gaussian mechanism to obtain the perturbed multi-modal spatio-temporal fusion embedding. The formula is as follows:

[0073] ;

[0074] ;

[0075] ;

[0076] ;

[0077] Where: Is the perturbed multi-modal spatio-temporal fusion embedding;

[0078] Is the noise vector;

[0079] Is the th element in the noise vector;

[0080] Is the Laplace-Gaussian function;

[0081] Is the sensitivity of the multi-modal fusion vector;

[0082] The number of other community users currently interacting with the user; The privacy budget allocated to the user;

[0083] Is the overparameter;

[0084] is a normalization parameter;

[0085] is an exponential function;

[0086] is the function probability density.

[0087] Furthermore, step two specifically includes: According to the multi-modal spatio-temporal fusion embedding of user perturbations, the intra-modal hierarchical attention mechanism is used to learn the attention weights of specific modalities, and then the inter-modal hierarchical attention mechanism is used to enhance the feature representation between multi-modalities, including the following steps:

[0088] Step 2.1: The edge node receives the perturbed multi-modal spatio-temporal fusion embeddings of multiple users, and then the edge node, according to the interacting users , learns the intra-modal attention weights of users and users under specific modalities through the intra-modal hierarchical attention mechanism, and then weights the multi-modal embeddings of users in the neighbor set to generate an intra-modal fusion vector, and the formula is as follows:

[0089] ;

[0090] ;

[0091] where: is the intra-modal attention weight of users and users under specific modalities;

[0092] is the query linear transformation matrix of specific modality ;

[0093] is the key-value linear transformation matrix of specific modality ;

[0094] is the perturbed multi-modal fusion embedding of user ;

[0095] is the perturbed multi-modal fusion embedding of user ;

[0096] is an exponential function;

[0097] is the set of users interacted by user ;

[0098] For the user 's intra-modal fusion vector;

[0099] Step 2.2: According to the fusion vector generated by the user in a specific modality Calculate the inter-modal weight value using the inter-modal attention mechanism, and then according to the inter-modal attention weight, generate the user's final inter-modal embedding vector by weighted summation of the fusion vectors of all modalities. The formula is as follows: ;

[0100] ;

[0101] ;

[0102] Where: is the inter-modal weight value;

[0103] is the exponential function;

[0104] is the learnable parameter matrix;

[0105] is the number of modality types;

[0106] For the user 's intra-modal fusion vector;

[0107] For the user 's final inter-modal embedding vector;

[0108] Step 2.3: According to the final inter-modal embedding vector, extract spatial features through spatial mean pooling, extract temporal features through temporal dimension pooling, and calculate the spatio-temporal correlation features of multiple users using the multi-head attention mechanism. The formula is as follows:

[0109] ;

[0110] Where: is the spatio-temporal correlation feature;

[0111] is the multi-head attention mechanism;

[0112] is the spatial linear transformation parameter;

[0113] is the temporal linear transformation parameter;

[0114] is the parameter of modal linear transformation;

[0115] is the user spatial feature of the final inter-modal embedding vector;

[0116] is the user temporal feature of the final inter-modal embedding vector;

[0117] is the user final inter-modal embedding vector;

[0118] Finally, a multi-user multi-modal spatio-temporal correlation matrix is obtained, where is the number of edge node users, is the data dimension.

[0119] Furthermore, step 3 specifically includes: adding Gaussian noise with adaptive dynamic decay based on gradient correlation to the training gradient of each round of the edge model to achieve correlation privacy protection of the model gradient, including the following steps:

[0120] Step 3.1: Based on the multi-user multi-modal spatio-temporal correlation matrix, use the time decay prediction loss function and the root mean square error loss function to pre-train through the edge local graph neural network model, and then calculate the model gradient according to the total prediction loss. The formula is as follows:

[0121] ;

[0122] ;

[0123] ;

[0124] ;

[0125] Where: is the time decay prediction loss function;

[0126] is the median of the time decay prediction loss function;

[0127] is the root mean square error loss function;

[0128] is the median of the root mean square error loss function;

[0129] is the total loss;

[0130] is the user For other community users At time The predicted weight;

[0131] Is the true weight;

[0132] Is the time decay factor;

[0133] Is the transpose operation;

[0134] Is the norm square calculation;

[0135] For the user All predicted weight vectors;

[0136] Is the regularization term;

[0137] Is the set of predicted user weights;

[0138] Is the model gradient;

[0139] Is the gradient operator.

[0140] Step 3.2: According to the modal gradient calculated in the Round, perform clipping based on the maximum sensitivity, and the formula is as follows:

[0141] ;

[0142] Where: Is the clipped gradient in the Round;

[0143] Is the Modal gradient calculated in the round;

[0144] Is The maximum sensitivity of;

[0145] Is Norm calculation;

[0146] Step 3.3: Based on the gradient spatio-temporal correlation protection method with adaptive attenuation, add adaptive Gaussian noise that satisfies Differential privacy to the clipped gradient, and the formula is as follows:

[0147] ;

[0148] ;

[0149] Wherein: is the noise scale of the round;

[0150] is the initial noise scale;

[0151] is the spatio-temporal correlation between the model gradient of the round and the model gradient of the round;

[0152] is the attenuation coefficient;

[0153] is the exponential function;

[0154] is the model gradient of the round of perturbation;

[0155] is the round of clipped gradient;

[0156] is Gaussian noise with a mean of 0 and a variance of ;

[0157] Step 3.4: All edge nodes submit the perturbed gradients to the cloud server. The cloud server aggregates all the perturbed gradients for global parameter optimization and returns the optimized parameters to each edge node for edge graph neural network update.

[0158] Furthermore, the fourth step specifically includes: extracting specific attribute relationships through relationship decoupling, helping the model capture the relationships between users - communities and users - users through memory - enhanced encoding, and finally performing multi - modal community discovery through a graph neural network spatio - temporal community discovery method, including the following steps:

[0159] Step 4.1: Based on the multi - user multi - modal spatio - temporal correlation matrix and the optimized parameters, perform relationship decoupling using the graph neural network spatio - temporal community discovery method, and the formula is as follows:

[0160] ;

[0161] ;

[0162] Wherein: is the user - community feature information;

[0163] is the user - user feature information;

[0164] The linear transformation matrix for user-community;

[0165] The linear transformation matrix for user-user;

[0166] The spatio-temporal correlation feature;

[0167] Step 4.2: According to the obtained user-community and user-user feature information, the features of user-community and user-user relationships are strengthened through the community memory unit and the user memory unit respectively, and the formula is as follows:

[0168] ;

[0169] ;

[0170] Where: The community memory unit;

[0171] The user memory unit;

[0172] The self-attention mechanism;

[0173] The user-community feature information;

[0174] The user-user feature information;

[0175] Step 4.3: According to the user-community feature information and the community memory unit, the user-community relationship embedding is calculated through weighted aggregation, and then according to the user-user feature information and the user memory unit, the user-user relationship embedding is calculated through weighted aggregation, and the formula is as follows:

[0176] ;

[0177] ;

[0178] Where: The user-community relationship embedding;

[0179] The Leaky ReLU activation function;

[0180] The community aggregation function;

[0181] The predicted set of user weights;

[0182] is a community memory unit;

[0183] is a user memory unit;

[0184] is user-community feature information;

[0185] is user-user feature information;

[0186] is user-user relationship embedding;

[0187] is the user 's neighbor set;

[0188] Step 4.4: According to the user-community relationship embedding and user-user relationship embedding, predict the score of the user for other users to perform community discovery. The formula is as follows:

[0189] ;

[0190] where: is the predicted score of the user for other users at time ;

[0191] are multi-layer perceptron parameters;

[0192] are learnable weight parameters;

[0193] is a concatenation operation;

[0194] is the user-community relationship embedding;

[0195] is the user-user relationship embedding;

[0196] Step 4.5: Perform iterative optimizations on the edge graph neural network model to minimize the loss function and obtain the optimal prediction weights for other community users; when the optimal prediction weights of other community users are greater than or equal to the weight threshold, it is considered that the community to which the user belongs is the user 's interest community, and send the community information to the user .

[0197] The second aspect of the present invention provides a community discovery device for multi-modal spatio-temporal correlation privacy protection, including a local multi-modal community user data spatio-temporal fusion protection module, a multi-level multi-modal user association enhancement module, an adaptive gradient spatio-temporal correlation protection module, a graph neural network intelligent community discovery module, a personal mobile device, an edge node, and a cloud server;

[0198] The local multi-modal community user data spatio-temporal fusion protection module performs spatio-temporal fusion on the local multi-modal community user data to protect the fused multi-modal spatio-temporal embedding;

[0199] The multi-level multi-modal user association enhancement module constructs a spatio-temporal association enhancement matrix based on intra-modal and inter-modal, enhancing the representation of multi-user interaction features;

[0200] The adaptive gradient spatio-temporal correlation protection module performs adaptive perturbation on the modal gradient based on gradient spatio-temporal correlation to achieve gradient spatio-temporal correlation protection;

[0201] The graph neural network intelligent community discovery module decouples the relationships and enhances the memory relationship encoding of the multi-level multi-modal user association matrix, predicting the community groups of interest to users based on user-community relationship embedding and user-user relationship embedding;

[0202] The personal mobile device is used to effectively extract the spatio-temporal correlation of multi-modal community user data based on the multi-modal spatio-temporal contrast learning privacy protection method to avoid the interference of modal relaxation, adding Laplace-Gaussian noise to protect the spatio-temporal correlation privacy;

[0203] The edge node is used to implement the multi-level multi-modal user association enhancement method, constructing a spatio-temporal association enhancement matrix based on intra-modal and inter-modal according to the multi-modal fusion embedding of multiple users;

[0204] The cloud server is used to aggregate all global perturbation gradients for global parameter optimization and return the optimized parameters to each edge node for edge graph neural network model update.

[0205] In the second aspect, the present application also provides a computing device, which has the function of implementing the method described in the first aspect above. The beneficial effects can be referred to the description of the first aspect and will not be elaborated here. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the structure of the device includes an acquisition module and a training module. Optionally, a construction module may also be included. These modules can implement the functions of the training node in the method example of the first aspect above. For specific reference, see the detailed description in the method example and will not be elaborated here.

[0206] In a third aspect, the present application further provides a computing device for implementing the functions of the method described in the first aspect above. The beneficial effects can be referred to the description of the first aspect and will not be elaborated here. The structure of the computing device includes a processor and a memory. The memory is used to store instructions and / or data. The memory is coupled to the processor. When the processor executes the program instructions stored in the memory, it can implement the functions of the training nodes in the examples of the first aspect. The structure of the computing device further includes a communication interface for communicating with other devices.

[0207] In a fourth aspect, the present application further provides a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to execute the methods in the first aspect and all possible designs of the first aspect.

[0208] In a fifth aspect, the present application further provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the methods in the first aspect and all possible designs of the first aspect.

[0209] In a sixth aspect, the present application further provides a computing chip connected to a memory. The chip is used to read and execute the software program stored in the memory and execute the methods in the first aspect and all possible implementation manners of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0210] In order to more clearly illustrate the technical solutions in the embodiments of the present drawings or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present drawings. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on the structures shown in these drawings.

[0211] Figure 1 It is a flowchart of the steps of a community discovery method for multi-modal spatio-temporal correlation privacy protection of the present invention;

[0212] Figure 2 It is a comparison chart of data availability between the embodiments of the present invention and traditional community user privacy protection methods under different total privacy budgets;

[0213] Figure 3 It is a comparison chart of data availability between the embodiments of the present invention and traditional gradient privacy protection methods under different edge model gradient privacy budgets;

[0214] Figure 4 It is a comparison chart of data availability between the embodiments of the present invention and traditional gradient privacy protection methods under different edge initial noise scales;

[0215] Figure 5 The figure shows the comparison of the hit rates between the embodiments of the present invention and traditional community discovery methods under different training batches;

[0216] The realization, functional features and advantages of this attached figure will be further described in combination with the embodiments with reference to the attached drawings. Detailed implementation manners

[0217] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described and explained below in combination with the attached drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0218] Obviously, the attached drawings in the following description are only some examples or embodiments of the present application. For those of ordinary skill in the art, without creative efforts, the present application can also be applied to other similar scenarios based on these drawings. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some designs, manufacturing or production changes based on the technical content disclosed in the present application are only conventional technical means and should not be understood that the content disclosed in the present application is insufficient.

[0219] If there is no special instruction, all implementation manners and optional implementation manners of the present application can be combined with each other to form a new technical solution.

[0220] If there is no special instruction, all technical features and optional technical features of the present application can be combined with each other to form a new technical solution.

[0221] If there is no special instruction, all steps of the present application can be carried out in sequence or randomly, and preferably in sequence. For example, the method includes steps (a) and (b), which means that the method may include steps (a) and (b) carried out in sequence, or may include steps (b) and (a) carried out in sequence. For example, it is mentioned that the method may further include step (c), which means that step (c) can be added to the method in any order. For example, the method may include steps (a), (b) and (c), or may include steps (a), (c) and (b), or may also include steps (c), (a) and (b), etc.

[0222] Unless otherwise specified, the terms "comprising" and "including" mentioned in this application are open-ended and can also be closed-ended. For example, the terms "comprising" and "including" can mean that other components not listed may also be included or comprised, or it can mean that only the listed components are included or comprised.

[0223] Unless otherwise specified, in this application, the term "or" is inclusive. For example, the phrase "A or B" means "A, B, or both A and B". More specifically, any of the following conditions satisfies the condition "A or B": A is true (or exists) and B is false (or does not exist); A is false (or does not exist) while B is true (or exists); or both A and B are true (or exist).

[0224] To better understand the solutions of the embodiments of this application, some related terms and concepts that may be involved in the embodiments of this application are introduced below.

[0225] (1) Differential Privacy is a mathematical framework for protecting personal privacy. It ensures that individual information is not leaked by adding random noise to the data while keeping the statistical characteristics of the dataset unchanged. The core idea of differential privacy is to ensure that when adding or deleting an individual in the dataset, the impact on the output result is minimal. Specifically, differential privacy requires that the distribution difference of the query results of two adjacent datasets (i.e., datasets that differ by only one individual) after randomization does not exceed a preset privacy budget. This mechanism can effectively prevent attackers from inferring individual information by analyzing the query results.

[0226] (2) Spatiotemporal features refer to features that consider both the time dimension and the space dimension in trajectory data. Spatiotemporal features are usually used to analyze and protect personal privacy information in trajectory data. For example, in trajectory data, by combining time features and space features, user privacy information can be protected more effectively. The research on spatiotemporal features usually includes dynamically and adaptively allocating weights to the time and space dimensions to ensure that the trajectory subflows meet the requirements of differential privacy.

[0227] (3) Laplace-Gaussian noise is a noise model that combines the characteristics of the Laplace distribution and the Gaussian distribution. In differential privacy, the Laplace mechanism is usually used to add noise to the data to protect privacy, while the Gaussian mechanism achieves a similar effect by adding random noise with a Gaussian distribution. Laplace-Gaussian noise is a noise model that combines the characteristics of the Laplace distribution and the Gaussian distribution and may be used for modeling more complex noise environments or specific applications in differential privacy.

[0228] (4) A Graph Neural Network (GNN) is a deep learning model specifically designed to process graph-structured data. Graph-structured data consists of nodes (or vertices) and edges (or connections). Nodes typically represent entities, while edges represent the relationships between entities. This data form exists widely in real life, such as social networks, recommendation systems, biological networks, etc. The core idea is to perform feature learning and reasoning by capturing the complex relationships between nodes in the graph. Different from traditional neural networks that process Euclidean space data, GNNs can directly operate on non-Euclidean spaces (i.e., graph structures), thus effectively processing the features of nodes, edges, and the entire graph in the graph.

[0229] In this embodiment, as Figure 1 shown, a community discovery method for multi-modal spatio-temporal correlation privacy protection includes the following steps:

[0230] Step 1: Based on the multi-modal spatio-temporal contrast learning privacy protection method to avoid modal relaxation, extract the spatio-temporal correlation of the multi-modal data of local community users, avoid the interference of modal relaxation and fuse multi-modal embeddings, and add Laplace-Gaussian noise to protect the privacy of the spatio-temporal correlation of the multi-modal fusion embeddings;

[0231] Multi-modal data refers to a data set containing various types of information, such as images, texts, audio, etc.

[0232] Laplace-Gaussian noise is a mixed noise model that combines the characteristics of the Laplace distribution and the Gaussian distribution. The Laplace distribution has a sharp peak, while the Gaussian distribution is relatively smooth. This noise model is often used to represent complex noise environments, especially in image processing and signal processing, and is widely used in differential privacy because it can provide stronger privacy protection.

[0233] Step 2: Based on the multi-level multi-modal user association enhancement method, construct a spatio-temporal association enhancement matrix based on intra-modal and inter-modal according to the multi-modal fusion embeddings of multi-community users, so as to enhance the representation of user interaction features;

[0234] Step 3: Based on the gradient spatio-temporal correlation protection method with adaptive attenuation, introduce Gaussian noise with adaptive attenuation to protect the privacy of spatio-temporal gradients;

[0235] Step 4: Based on the spatio-temporal community discovery method of graph neural networks, deeply mine the spatio-temporal interaction features between users and communities, and between users and users, and perform multi-modal community discovery.

[0236] Furthermore, a privacy protection method for multi-modal spatio-temporal contrast learning based on avoiding modal relaxation is specifically to alternately learn the unimodal information of local community users through random probability, extract the spatio-temporal correlation between modalities using a deep graph information encoder, and fuse multi-modal feature embeddings based on the modal information and spatio-temporal correlation features.

[0237] Furthermore, Step 1 includes the following steps:

[0238] Step 1.1: Given the personal text modal data contained by the user and the numerical weight data of interacting with other community users, randomly select any modality for alternate learning according to random probability;

[0239] Step 1.2: Use a deep graph information encoder to extract the spatio-temporal correlation between modalities, and optimize the learning result by minimizing the prediction risk function of this modality. The formula is as follows:

[0240] ;

[0241] ;

[0242] During the extraction and learning process of spatio-temporal correlation, ensure that the gradient update direction is orthogonal to the prior modal feature direction through gradient alignment to avoid the modal relaxation problem in model learning. The formula is as follows:

[0243] ;

[0244] ;

[0245] ;

[0246] Step 1.3: Based on the learned unimodal information and the spatio-temporal correlation between modalities, fuse multi-modal feature information through weighted contrast learning. The formula is as follows:

[0247] ;

[0248] ;

[0249] Step 1.4: According to the fused multi-modal spatio-temporal fusion embedding, protect the spatio-temporal correlation privacy of the fusion embedding through the Laplace-Gaussian mechanism. First, calculate the importance parameter of the user in the social network community according to the number of other community users interacted by the user, and dynamically allocate the privacy budget of this user according to the importance parameter. The formula is as follows:

[0250] ;

[0251] ;

[0252] Then, based on the Laplace-Gaussian mechanism, perturb the multi-modal spatio-temporal fusion embedding of the user to obtain the perturbed multi-modal spatio-temporal fusion embedding. The formula is as follows:

[0253] ;

[0254] ;

[0255] ;

[0256] ;

[0257] Furthermore, step two specifically includes: according to the perturbed multi-modal spatio-temporal fusion embedding of the user, adopt the intra-modal hierarchical attention mechanism to learn the attention weights of specific modalities, and then adopt the inter-modal hierarchical attention mechanism to enhance the feature representation between multi-modalities, including the following steps:

[0258] Step 2.1: The edge node receives the perturbed multi-modal spatio-temporal fusion embeddings of multiple users. Then, the edge node, according to the users interacting users , through the intra-modal hierarchical attention mechanism, learns the intra-modal attention weights of the users and the users under specific modalities. Then, in the neighbor set, weight the multi-modal embeddings of the user to generate the intra-modal fusion vector. The formula is as follows:

[0259] ;

[0260] ;

[0261] Step 2.2: According to the fusion vector generated by the user in the specific modality , adopt the inter-modal attention mechanism to calculate the inter-modal weight value. Then, according to the inter-modal attention weights, by weighted summing the fusion vectors of all modalities, generate the final inter-modal embedding vector of the user . The formula is as follows:

[0262] ;

[0263] ;

[0264] Step 2.3: According to the final inter-modal embedding vector, extract spatial features through spatial mean pooling, extract temporal features through temporal dimension pooling, and adopt the multi-head attention mechanism to calculate the spatio-temporal correlation features of multiple users. The formula is as follows:

[0265] ;

[0266] Finally, a multi - user multi - modal spatio - temporal correlation matrix is obtained , where is the number of users of edge nodes, is the data dimension.

[0267] Furthermore, step three specifically includes: adding Gaussian noise with adaptive dynamic decay based on gradient correlation to the gradient of each round of training of the edge model to achieve correlation privacy protection of the model gradient, including the following steps:

[0268] Step 3.1: Based on the multi - user multi - modal spatio - temporal correlation matrix, using a time - decay prediction loss function and a root - mean - square error loss function, pre - train through the edge local graph neural network model, and then calculate the model gradient according to the total prediction loss. The formula is as follows:

[0269] ;

[0270] ;

[0271] ;

[0272] ;

[0273] Step 3.2: According to the modal gradient calculated in the round, perform clipping based on the maximum sensitivity. The formula is as follows:

[0274] ;

[0275] Step 3.3: In this embodiment, based on the gradient spatio - temporal correlation protection method with adaptive decay, first capture the spatio - temporal features between and through a convolutional - recurrent neural network. The formula is as follows:

[0276] ;

[0277] Where: is the convolutional - recurrent neural network function;

[0278] is the multi - layer perceptron;

[0279] are the convolutional - recurrent neural network feature extraction parameters;

[0280] is the clipped gradient in the round;

[0281] is the Round - cropped gradient;

[0282] According to and the spatio - temporal features between, calculate the spatio - temporal mutual information to obtain and the spatio - temporal correlation between, the formula is as follows:

[0283] ;

[0284] Where: is and the spatio - temporal correlation between;

[0285] is the supremum,

[0286] is the expectation;

[0287] is the convolutional - recurrent neural network function;

[0288] is the round and the round - cropped gradient joint distribution;

[0289] is the round - cropped marginal distribution;

[0290] is the round - cropped marginal distribution;

[0291] is the round - cropped gradient;

[0292] is the round - cropped gradient;

[0293] According to the calculated spatio - temporal correlation, add adaptive Gaussian noise that satisfies to the cropped gradient - differential privacy, the expression is:

[0294] ;

[0295] ;

[0296] Step 3.4: All edge nodes submit the perturbed gradients to the cloud server. The cloud server aggregates all the perturbed gradients for global parameter optimization and returns the optimized parameters to each edge node for edge graph neural network update.

[0297] Furthermore, Step 4 specifically includes: extracting specific attribute relationships through relationship decoupling, helping the model capture the relationships between users and communities, and users and users through memory-enhanced encoding, and finally performing multi-modal community discovery through a graph neural network spatio-temporal community discovery method, including the following steps:

[0298] Step 4.1: Based on the multi-user multi-modal spatio-temporal correlation matrix and optimization parameters, perform relationship decoupling through a graph neural network spatio-temporal community discovery method. The formula is as follows:

[0299] ;

[0300] ;

[0301] Step 4.2: According to the obtained user-community and user-user feature information, strengthen the features of the user-community and user-user relationships through the community memory unit and the user memory unit respectively. The formula is as follows:

[0302] ;

[0303] ;

[0304] Step 4.3: According to the user-community feature information and the community memory unit, calculate the user-community relationship embedding through weighted aggregation, and then according to the user-user feature information and the user memory unit, calculate the user-user relationship embedding through weighted aggregation. The formula is as follows:

[0305] ;

[0306] ;

[0307] Step 4.4: According to the user-community relationship embedding and the user-user relationship embedding, predict the rating for other community users to perform community discovery. The formula is as follows:

[0308] ;

[0309] Step 4.5: Perform iterative optimization on the edge graph neural network model to minimize the loss function and obtain the optimal prediction weights for other community users; when the optimal prediction weights of other community users are greater than or equal to the weight threshold, it is considered that the community to which the user belongs is the interest community of user and send the community information to user .

[0310] The second aspect of the present application provides a community discovery device for multi-modal spatio-temporal correlation privacy protection, including a local multi-modal community user data spatio-temporal fusion protection module, a multi-level multi-modal user association enhancement module, an adaptive gradient spatio-temporal correlation protection module, a graph neural network intelligent community discovery module, a personal mobile device, an edge node, and a cloud server;

[0311] The local multi-modal community user data spatio-temporal fusion protection module performs spatio-temporal fusion on the local multi-modal community user data to protect the fused multi-modal spatio-temporal embedding;

[0312] The multi-level multi-modal user association enhancement module constructs a spatio-temporal association enhancement matrix based on intra-modal and inter-modal to enhance the representation of multi-user interaction features;

[0313] The adaptive gradient spatio-temporal correlation protection module performs adaptive perturbation on the modal gradient based on gradient spatio-temporal correlation to achieve gradient spatio-temporal correlation protection;

[0314] The graph neural network intelligent community discovery module decouples the relationship of the multi-level multi-modal user association matrix and enhances the memory relationship encoding, and predicts the community groups of interest to users based on user-community relationship embedding and user-user relationship embedding;

[0315] The personal mobile device is used for a multi-modal spatio-temporal contrast learning privacy protection method based on avoiding modal relaxation, effectively extracts the spatio-temporal correlation of multi-modal community user data to avoid the interference of modal relaxation, adds Laplace-Gaussian noise, and protects the spatio-temporal correlation privacy;

[0316] The edge node is used to implement a multi-level multi-modal user association enhancement method, and constructs a spatio-temporal association enhancement matrix based on intra-modal and inter-modal according to the multi-modal fusion embedding of multiple users;

[0317] The cloud server is used to aggregate all global perturbation gradients for global parameter optimization, and returns the optimized parameters to each edge node for edge graph neural network model update.

[0318] To protect the privacy of community user multi-modal data and model gradients, it is theoretically proven that a community discovery method for multi-modal spatio-temporal correlation privacy protection proposed by the present invention satisfies differential privacy.

[0319] Proof: Since the perturbed multi-modal fusion embedding of community users is added with noise that satisfies the Laplace-Gaussian distribution, where the privacy budget is allocated as , based on the properties of differential privacy, the multi-modal spatio-temporal contrast learning privacy protection method based on avoiding modal relaxation in this instance satisfies differential privacy.

[0320] Because the model perturbation gradient is added with noise that satisfies the Gaussian mechanism, where the privacy budget is allocated as and, based on the properties of relaxed differential privacy, this example is based on an adaptive decay gradient spatio-temporal correlation protection method that satisfies differential privacy. Based on the serial combination principle of differential privacy, a community discovery method for multi-modal spatio-temporal correlation privacy protection proposed by the present invention satisfies

[0321] differential privacy, where and realizes the privacy protection of multi-modal fusion embedding of local community users and model gradients. In this embodiment, based on the real movie rating dataset (Ciao), different parameters are adopted: the privacy budget for local multi-modal fusion embedding

[0322] , the privacy budget for edge model gradients , the initial noise scale of the edge , and the training batch Epoch are used to evaluate the usability of the privacy protection of the perturbed data by the present invention. The results of the comparative experiments are as shown, where Figures 2 to 5 is a comparison graph of the data availability between the embodiments of the present invention and traditional community user privacy protection methods under different total privacy budgets; Figure 2 is a comparison graph of the data availability between the embodiments of the present invention and traditional gradient privacy protection methods under different edge model gradient privacy budgets; Figure 3 is a comparison graph of the data availability between the embodiments of the present invention and traditional gradient privacy protection methods under different edge initial noise scales; Figure 4 is a comparison graph of the hit rate between the embodiments of the present invention and traditional community discovery methods under different training batches. Figure 5

[0323] It can be seen from each comparison graph that a community discovery method for multi-modal spatio-temporal correlation privacy protection of the present application can well achieve the effect of privacy protection for multi-modal data when discovering community groups of interest to community users, and ensure the security of multi-modal data.

[0324] It should be noted that the present application is not limited to the above embodiments. The above embodiments are only examples, and embodiments with the same composition and the same effect as the technical idea within the scope of the technical solution of the present application are included in the technical scope of the present application. In addition, within the scope of not departing from the gist of the present application, various modifications that can be thought of by those skilled in the art to the embodiments and other ways constructed by combining some constituent elements of the embodiments are also included in the scope of the present application.​

Claims

1. A multimodal spatiotemporal correlation privacy-preserving community discovery method, characterized in that: The following steps are involved: Step 1: Based on the multimodal spatiotemporal contrast learning privacy protection method to avoid modal relaxation, the spatiotemporal correlation of multimodal data of local community users is extracted to avoid the interference of modal relaxation and fuse multimodal embeddings, and Laplace-Gaussian noise is added to protect the privacy of spatiotemporal correlation of multimodal fusion embeddings; The multimodal spatiotemporal contrastive learning privacy protection method based on avoiding modal relaxation specifically learns the single modal information of local community users through random probability alternation, uses a deep graph information encoder to extract the spatiotemporal correlation between modalities, and fuses multimodal feature embedding according to modal information and spatiotemporal correlation features; Step 2: Based on the multi-level multimodal user association enhancement method, according to the multimodal fusion embedding of multi-community users, a spatiotemporal association enhancement matrix based on intra-modal and inter-modal relationships is constructed to enhance the representation of user interaction features; Step 3: Based on the adaptive attenuation gradient spatiotemporal correlation protection method, the adaptive attenuation Gaussian noise is introduced to protect the privacy of spatiotemporal gradients; Step 4: Based on the spatiotemporal community discovery method of graph neural network, we deeply explore the spatiotemporal interaction characteristics between users and communities, and between users, and conduct multimodal community discovery.

2. According to claim 1, a multimodal spatiotemporal correlation privacy-preserving community discovery method is characterized in that: The step 1 comprises the following steps: Step 1.1: Given the personal text modality data of the user and the numerical weight data of other community users in interaction, select any modality for alternating learning according to random probability; Step 1.2: Use the deep image information encoder to extract the spatiotemporal correlation between modalities and optimize the learning results by minimizing the prediction risk function of the modality; In the process of extracting and learning spatiotemporal correlation, gradient alignment is used to ensure that the gradient update direction is orthogonal to the direction of the prior modal feature, thus avoiding the modal relaxation problem in model learning. Step 1.3: Based on the learned single-modal information and the spatiotemporal correlation between modalities, multimodal feature information is fused through weighted contrast learning; Finally, multimodal spatiotemporal fusion embedding is obtained through weighted multimodal fusion , is the data dimension, is the Lth eigenvalue of the multimodal spatiotemporal fusion embedding; Step 1.4: Based on the fused multimodal spatiotemporal fusion embedding, the Laplace-Gaussian mechanism is used to protect the spatiotemporal correlation privacy of the fused embedding. First, the importance parameter of the user in the social network community is calculated according to the number of other community users the user interacts with, and the privacy budget of the user is dynamically allocated according to the importance parameter; Then, based on the Laplace-Gaussian mechanism, the user's multimodal spatiotemporal fusion embedding is perturbed to obtain the perturbed multimodal spatiotemporal fusion embedding.

3. The method for discovering communities with privacy protection based on multimodal spatiotemporal correlation according to claim 1, characterized in that: The step 2 specifically includes: according to the multimodal spatiotemporal fusion embedding of the user disturbance, using the intra-modal hierarchical attention mechanism to learn the attention weight of a specific modality, and then using the inter-modal hierarchical attention mechanism to enhance the feature representation between the multimodalities, including the following steps: Step 2.1: The edge node receives the perturbed multimodal spatiotemporal fusion embedding of multiple users, and then the edge node Interactive users , learning user and users Intra-modality attention weights in a specific modality are then used to classify users in the neighbor set. The multimodal embedding of is weighted to generate an intra-modal fusion vector; Step 2.2: According to the user In a specific mode The fusion vector generated in , uses the inter-modal attention mechanism to calculate the inter-modal weight value, and then generates the user by weighted summing up the fusion vectors of all modalities according to the inter-modal attention weight. The final inter-modal embedding vector of ; Step 2.3: Based on the final inter-modal embedding vector, spatial features are extracted through spatial mean pooling, temporal features are extracted through temporal dimension pooling, and multi-head attention mechanism is used to calculate multi-user spatiotemporal correlation features; Finally, the multi-user multi-modal spatiotemporal correlation matrix is ​​obtained ,in is the number of edge node users, is the data dimension.

4. The method for discovering communities with privacy protection based on multimodal spatiotemporal correlation according to claim 1, characterized in that: The step three specifically includes: adding adaptive dynamically attenuated Gaussian noise based on gradient correlation to each round of training gradient of the edge model to achieve correlation privacy protection of the model gradient, including the following steps: Step 3.1: Based on the multi-user multi-modal spatiotemporal correlation matrix, the time decay prediction loss function and the root mean square error loss function are used to pre-train the edge local graph neural network model, and then the model gradient is calculated according to the total prediction loss; Step 3.2: According to The modal gradients calculated in rounds are clipped based on maximum sensitivity; Step 3.3: Based on the adaptive attenuation gradient spatiotemporal correlation protection method, add the gradient after clipping to meet Adaptive Gaussian noise for differential privacy; Step 3.4: All edge nodes submit the perturbation gradients to the cloud server, which aggregates all perturbation gradients for global parameter optimization and returns the optimized parameters to each edge node for edge graph neural network update.

5. The method for discovering communities with privacy protection based on multimodal spatiotemporal correlation according to claim 1, characterized in that: The step 4 specifically includes: extracting specific attribute relationships through relationship decoupling, helping the model capture the relationship between users and communities and users and users through memory enhancement coding, and finally performing multimodal community discovery through a spatiotemporal community discovery method based on a graph neural network, including the following steps: Step 4.1: According to the multi-user multi-modal spatiotemporal correlation matrix and optimization parameters, the spatiotemporal community discovery method based on graph neural network is used to perform relationship decoupling; Step 4.2: Based on the obtained user-community and user-user feature information, the features of the user-community and user-user relationships are strengthened through the community memory unit and the user memory unit respectively; Step 4.3: Calculate the user-community relationship embedding by weighted aggregation based on the user-community feature information and the community memory unit, and then calculate the user-user relationship embedding by weighted aggregation based on the user-user feature information and the user memory unit; Step 4.4: Based on the user-community relationship embedding and user-user relationship embedding, predict the user For other community users scores for community discovery; Step 4.5: Edge Graph Neural Network Model Iterate optimization to minimize the loss function and obtain the optimal prediction weight for other community users; when the optimal prediction weight of other community users is greater than or equal to the weight threshold, the community to which the user belongs is considered to be the user interest communities, and send community information to users .

6. A multimodal spatiotemporal correlation privacy-preserving community discovery device, characterized in that: It includes local multimodal community user data spatiotemporal fusion protection module, multi-level multimodal user association enhancement module, adaptive gradient spatiotemporal correlation protection module, graph neural network intelligent community discovery module, personal mobile devices, edge nodes and cloud servers; The local multimodal community user data spatiotemporal fusion protection module performs spatiotemporal fusion on the local multimodal community user data and protects the fused multimodal spatiotemporal embedding; The multi-level multi-modal user association enhancement module constructs a spatiotemporal association enhancement matrix based on intra-modality and inter-modality to enhance the representation of multi-user interaction features; The adaptive gradient spatiotemporal correlation protection module performs adaptive perturbation on the modal gradient based on the gradient spatiotemporal correlation to achieve gradient spatiotemporal correlation protection; The graph neural network intelligent community discovery module performs relationship decoupling and memory relationship enhancement encoding on the multi-level multi-modal user association matrix, and predicts the community groups that users are interested in based on user-community relationship embedding and user-user relationship embedding; The personal mobile device is used to effectively extract the spatiotemporal correlation of multimodal community user data to avoid the interference of modal relaxation, add Laplace-Gaussian noise, and protect the privacy of spatiotemporal correlation based on a multimodal spatiotemporal contrast learning privacy protection method that avoids modal relaxation; The multimodal spatiotemporal contrastive learning privacy protection method based on avoiding modal relaxation specifically learns the single modal information of local community users through random probability alternation, uses a deep graph information encoder to extract the spatiotemporal correlation between modalities, and fuses multimodal feature embedding according to modal information and spatiotemporal correlation features; The edge node is used to implement a multi-level multi-modal user association enhancement method, and construct a spatiotemporal association enhancement matrix based on intra-modal and inter-modal based on multi-modal fusion embedding of multiple users; The cloud server is used to aggregate all global perturbation gradients for global parameter optimization, and return the optimized parameters to each edge node for edge graph neural network model update.

Images