Privacy protection method in multi-modal data fusion

Abstract

The invention discloses a privacy protection method in multi-modal data fusion, and the method comprises the following steps: S1, obtaining multi-modal original data of at least two modal types, carrying out the standardization preprocessing of each modal data, eliminating the differences of different modal data in dimensions and value ranges, and obtaining standardized multi-modal input data; s2, a privacy-semantic coupling model is constructed, the model comprises a privacy feature space and a semantic feature space, and the invention relates to the technical field of data security. According to the privacy protection method in multi-modal data fusion, a privacy-semantic coupling model is constructed, a privacy variable is used as an implicit interference factor to be embedded into a multi-modal fusion process, privacy noise in multi-modal data is automatically corrected and counteracted through a data processing model, and under the premise that original sensitive data is not explicitly exposed, the privacy of the multi-modal data is effectively protected. The privacy protection in the multi-modal data fusion process is realized, and the risk of user privacy disclosure is greatly reduced.

Description

Technical Field

[0001] This invention relates to the field of data security, and more specifically, to a privacy protection method in multimodal data fusion. Background Technology

[0002] With the rapid development of information technology, multimodal data, such as text, images, audio, and time-series data, has been widely used in various fields. Multimodal data fusion, by integrating data from different modalities, can fully utilize the advantages of each modality to provide more comprehensive and accurate information, thereby improving the reliability of data analysis and decision-making. For example, in the field of intelligent security, combining video surveillance images and audio information can more accurately identify abnormal events; in medical diagnosis, integrating patient medical records, medical images, and physiological time-series data helps improve diagnostic accuracy.

[0003] However, multimodal data often contains a large amount of sensitive information, such as personal identification information, privacy behavior data, and sensitive preference characteristics. If not handled properly during data fusion, this sensitive information can easily be leaked, posing serious privacy risks to users. Traditional privacy protection methods, such as data anonymization and encryption, have many limitations in multimodal data fusion scenarios. Data anonymization may significantly reduce data usability, affecting the fusion effect; while encryption methods have high computational complexity when processing multimodal data and struggle to balance the strength of privacy protection with data usability.

[0004] Furthermore, data from different modalities differ significantly in terms of units and value ranges. Direct fusion can be affected by these differences, impacting the accuracy and stability of the fusion results. Therefore, effectively protecting data privacy while ensuring the effectiveness of multimodal data fusion has become a critical issue that urgently needs to be addressed. Summary of the Invention

[0005] The purpose of this invention is to provide a privacy protection method in multimodal data fusion, which solves the problem that existing privacy protection methods, such as data anonymization and encryption, have many limitations in multimodal data fusion scenarios. Data anonymization may lead to a significant reduction in data availability and affect the fusion effect; while encryption methods have high computational complexity when processing multimodal data and it is difficult to balance the strength of privacy protection and data usability, thus failing to meet the usage requirements.

[0006] This invention achieves the above objective through the following technical solution: a privacy protection method in multimodal data fusion, comprising the following steps: S1. Obtain multimodal raw data of at least two modal types, perform standardization preprocessing on each modal data to eliminate differences in the units and value ranges of different modal data, and obtain standardized multimodal input data; S2. Construct a privacy-semantic coupling model, which includes a privacy feature space and a semantic feature space, establish a bidirectional mapping mechanism between the two, and embed privacy variables as implicit interference factors into the multimodal fusion process; S3. Based on the privacy-semantic coupling model, the privacy noise in multimodal data is automatically corrected and canceled through the data processing model, and mutual constraints are achieved by utilizing the coupling relationship between different modalities. S4. Without explicitly exposing the original sensitive data, restore the true semantic information of multimodal data through model decoupling and semantic reconstruction, and complete the multimodal data fusion under privacy protection.

[0007] Furthermore, the multimodal raw data includes: The data includes at least two of the following: text, image, audio, and time-series data. The standardization preprocessing includes using corresponding normalization methods for different modalities: word vector normalization for text data, pixel value normalization for image data, amplitude normalization for audio data, and Z-score normalization for time-series data. The preprocessed data is divided into training, validation and test sets according to a preset ratio. The division process ensures that the modality distribution and privacy attribute distribution of each dataset are consistent with the original data.

[0008] Furthermore, the steps for constructing a privacy-semantic coupling model include: A privacy feature space and a semantic feature space are constructed respectively. The privacy feature space is generated by a set of sensitive attributes. The sensitive attributes include at least one of personal identity information, privacy behavior data, and sensitive preference features. Each sensitive attribute corresponds to an independent privacy dimension. The semantic feature space is composed of the core semantic representations of each modality, and the semantic features of different modalities respectively reflect the core information of the corresponding modality; A coupling function between the privacy feature space and the semantic feature space is established, and a bidirectional mapping is achieved through a deep neural network. Privacy variables are embedded as implicit interference factors into the feature interaction process of multimodal fusion.

[0009] Furthermore, the deep neural network adopts an encoder-decoder architecture, with the encoder containing 3-6 Transformer encoder layers, each layer containing a multi-head attention mechanism, a feedforward neural network, and layer normalization, and the decoder using 2-4 Transformer decoder layers. The initial values ​​of the neural network weights are initialized using a Xavier normal distribution, the weight update rule is based on stochastic gradient descent, and the learning rate is set to a dynamic adjustment mode.

[0010] Furthermore, the steps for automatic correction and cancellation of privacy noise based on the privacy-semantic coupling model include: For each modality of standardized data, initial semantic features and privacy interference features are extracted using a feature extraction network; A privacy noise removal model is constructed. Taking the initial semantic features and privacy noise features as input, a deep learning network is used to learn the mapping relationship between privacy noise and semantic features, so as to achieve accurate cancellation of privacy noise. By leveraging the coupling constraints between different modalities, we can ensure the synergy of privacy noise cancellation during multimodal fusion and avoid semantic distortion caused by privacy de-processing of a single modality.

[0011] Furthermore, the feature extraction network includes: Modality-specific feature extraction network and privacy detection network. The modality-specific feature extraction network adopts the corresponding network structure for different modalities. The text modality adopts the encoding layer of the BERT model, the image modality adopts the intermediate layer output of ResNet50 or VisionTransformer, the audio modality adopts Mel spectral features + CNN network, and the temporal modality adopts LSTM network. The privacy detection network is a 2-3 layer fully connected network with a Sigmoid activation function, which can accurately identify and separate interference components related to sensitive attributes.

[0012] Furthermore, the coupling constraint relationship is constructed based on the semantic consistency between modalities, specifically by calculating the similarity between the semantic features of different modalities after privacy interference removal; A preset semantic consistency threshold is set. When the similarity does not meet the threshold requirement, the relevant model parameters are adjusted through backpropagation to ensure that the privacy-de-identified semantic features of different modalities maintain semantic alignment.

[0013] Furthermore, the steps to recover true semantic information through model decoupling and semantic reconstruction include: By using the decoupling mechanism of the privacy-semantic coupling model, semantic features and privacy interference features are separated. The decoupling process is achieved through adversarial training, and a privacy discriminator and a semantic discriminator are constructed. The privacy-semantic coupling model forms an adversarial relationship with the two discriminators. Based on the semantic features of each modality after privacy interference removal, feature fusion is performed through a multimodal fusion network, and a cross-modal attention mechanism is adopted to achieve adaptive weighted fusion of semantic features of different modalities; After generating a unified semantic representation, privacy-de-identified data for each modality is reconstructed using a modality-specific reconstruction network, and then integrated to obtain privacy-preserving multimodal fusion data.

[0014] Furthermore, the privacy discriminator is a 3-layer fully connected network plus a Softmax output layer, with 128-512 hidden layer neurons, used to distinguish whether the input features contain privacy interference components; The semantic discriminator uses a lightweight model consistent with the modality feature extraction network structure to determine whether the features retain the core semantic information of the original modality. During adversarial training, the discriminator and the generator model are trained alternately until the preset training stopping condition is met.

[0015] Furthermore, the attention weights of the multimodal fusion network are determined in the following way: The relevance score between each modality semantic feature and the fusion target is calculated, and the attention weight is obtained by dot product similarity calculation and then normalized by the Softmax function. The modality-specific reconstruction network employs corresponding reconstruction methods for different modalities: text reconstruction uses a Transformer decoder + vocabulary mapping, image reconstruction uses a convolutional transpose network, audio reconstruction uses LSTM + inverse spectral transform, and time-series data reconstruction uses a fully connected network.

[0016] The beneficial effects of this invention are as follows: 1. The constructed privacy-semantic coupling model embeds privacy variables as implicit interference factors into the multimodal fusion process. Through the data processing model, privacy noise in multimodal data is automatically corrected and canceled. Without explicitly exposing the original sensitive data, privacy protection is achieved in the multimodal data fusion process, which greatly reduces the risk of user privacy leakage.

[0017] 2. By decoupling the model and reconstructing the semantics, the true semantic information of multimodal data can be recovered. The coupling relationship between different modalities is used to achieve mutual constraints, avoiding the semantic distortion that may be caused by privacy processing of a single modality. This ensures that the fused data can still accurately reflect the core semantics of the original data, thus guaranteeing the usability and value of the data.

[0018] 3. The method proposed in this invention is applicable to raw data of various modalities such as text, images, audio, and time series data. It adopts corresponding standardized preprocessing methods, feature extraction networks, and reconstruction methods for different modalities of data, and has wide applicability and flexibility, which can meet the needs of multimodal data fusion in different fields and scenarios.

[0019] 4. By constructing a privacy-removing interference model, the mapping relationship between privacy noise and semantic features is accurately learned, thereby achieving precise cancellation of privacy noise; by utilizing a cross-modal attention mechanism, adaptive weighted fusion of semantic features from different modalities is achieved, which can fully leverage the advantages of each modality of data and improve the effect and accuracy of multimodal data fusion.

[0020] 5. A deep neural network with an encoder-decoder architecture is adopted, and its network layer number, weight initialization, weight update rules are optimized. At the same time, the model is decoupled through adversarial training, and the discriminator and the generator model are trained alternately, which helps to improve the training efficiency and performance of the model, enabling the model to better complete the multimodal data fusion task under privacy protection. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating the overall method of the present invention; Figure 2 This describes the privacy-semantic coupling model construction process of the present invention; Figure 3 This is a flowchart of the privacy noise correction and cancellation process of the present invention; Figure 4 This is a flowchart of the model decoupling and semantic reconstruction process of the present invention. Detailed Implementation

[0022] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0023] Example 1: Please see Figure 1-4 This invention provides a technical solution: a privacy protection method in multimodal data fusion, the method comprising: S1. Obtain the original multimodal data, preprocess each modal data separately, and obtain standardized multimodal input data; Multimodal raw data refers to the initial data set containing various modalities, such as text, images, audio, and video. These data come from different information sources and have different feature representations and characteristics. Preprocessing involves a series of operations on the raw data to improve data quality, unify data format, and extract useful features, making it more suitable for subsequent analysis and processing. Common preprocessing operations include data cleaning, data normalization, and data augmentation. Standardized multimodal input data is data that, after preprocessing, achieves a unified standard in terms of format, scale, and feature representation, facilitating unified processing and analysis by subsequent models. S2. Construct a privacy-semantic coupling model. The model includes a bidirectional mapping mechanism between the privacy feature space and the semantic feature space, and embeds privacy variables as implicit interference factors into the multimodal fusion process. The privacy-semantic coupling model is a specially designed architecture aimed at simultaneously processing privacy and semantic information in multimodal data. By establishing a certain relationship between the two, it achieves privacy protection without losing the semantic content of the data. The privacy feature space represents the space of privacy-related features in multimodal data. In this space, data features can reflect sensitive information about an individual, such as personal identity, health status, and financial information. The semantic feature space represents the space of semantically related features in multimodal data. Semantic features reflect the actual meaning, content, or concept conveyed by the data, such as the object category in an image or the topic of a text. The bidirectional mapping mechanism refers to establishing a connection between the privacy feature space and the semantic feature space. The mutual conversion mapping relationship, that is, mapping from the privacy feature space to the semantic feature space, and vice versa, helps to better understand and utilize the semantics of data while protecting privacy. Privacy variables are variables that represent privacy information in the model and are embedded as implicit interference factors in the multimodal fusion process. They do not appear directly and explicitly in the data, but play a role in interfering with and protecting privacy during data processing through model design and training. The multimodal fusion process is the process of integrating and combining data from different modalities to make full use of the advantages of each modality and obtain more comprehensive and accurate information. For example, fusing image and text data to better understand the content and context of the image. S3. Based on the privacy-semantic coupling model, the privacy noise in multimodal data is automatically corrected and canceled through regression or mapping models, and mutual constraints are achieved by utilizing the coupling relationship between different modalities. Among them, regression or mapping models are used to establish mathematical models of the relationship between input data (multimodal data) and output (which may be privacy-processed data or related features). Regression models are usually used to predict continuous value outputs, while mapping models are more widely used to transform data from one space to another. Here, these models are used to handle privacy noise in multimodal data. Privacy noise is privacy-related interference information or unnecessary fluctuations in multimodal data. This noise may be introduced during data acquisition, transmission, or storage and will negatively affect the privacy and semantic accuracy of the data. Automatic correction and cancellation use regression or mapping models to automatically identify and adjust privacy noise according to the inherent relationships and patterns between different modal data, so as to reduce or eliminate its impact on the data, thereby obtaining a more accurate and privacy-preserving data representation. The coupling relationship between different modalities is the mutual correlation and influence between different modal data. For example, when describing a scene, the visual information in the image and the descriptive information in the text are complementary and interrelated. Using this coupling relationship, when the data of one modality is interfered with by privacy noise, it can be constrained and corrected by the data of other modalities. S4. Without explicitly exposing the original sensitive data, restore the true semantic information of multimodal data through model decoupling and semantic reconstruction, and complete the multimodal data fusion under privacy protection. Among these steps, the first is to avoid explicitly exposing original sensitive data, ensuring that sensitive information contained in the original data is not presented in a direct or obvious way throughout the entire data processing and fusion process, thus preventing unauthorized access or leakage of sensitive information. The second is to decouple the data after it has been processed by the privacy-semantic coupling model from the state where privacy and semantics are interrelated, and extract information related to privacy and semantics separately. This step aims to further clarify and distinguish the different components in the data. The third is to reconstruct and restore the true semantic content of the multimodal data based on the extracted semantic information, on the basis of model decoupling. In this way, valuable information conveyed by the data can still be obtained while protecting privacy. The fourth is to achieve a data processing goal after completing all the above steps, which can effectively protect the privacy information in the multimodal data, and make full use of the advantages of each modality to achieve accurate and useful fusion results.

[0024] It should be noted that during use, the original data of each modality is preprocessed to obtain standardized input, laying a good foundation for subsequent processing, improving data quality and processing efficiency. A privacy-semantic coupling model is constructed, which cleverly balances privacy protection and semantic preservation through a bidirectional mapping mechanism and embedding privacy variables as implicit interference factors. This avoids the loss of key semantic information due to excessive privacy protection. Regression or mapping models are used to automatically correct and offset privacy noise, and the mutual constraints of coupling relationships between different modalities can effectively reduce the risk of privacy leakage and enhance data security. Without explicitly exposing the original sensitive data, the true semantics are restored through model decoupling and semantic reconstruction, completing the fusion. This satisfies the needs of data fusion and utilization while effectively protecting user privacy, finding a good balance between data utilization and privacy protection.

[0025] In one embodiment, multimodal raw data is acquired, and preprocessing is performed on each modal data separately, including: Multimodal raw data includes at least two of the following: text, images, audio, and time-series data, denoted as dataset: in, Indicates the number of modal types. Indicates the first The raw data of each modality, Text data can come from natural language text, structured text fields, etc.; image data can include still images, images extracted from video frames, etc.; audio data covers speech signals, environmental audio, etc.; and time series data includes sensor data and behavior recording data in time series format. Standardize each modal data to eliminate differences in units and value ranges between different modal data, and ensure the consistency of multimodal fusion; The text data is normalized using word vector normalization. The text is converted into fixed-dimensional word vectors of 512 or 768 dimensions through pre-trained word embedding models such as Word2Vec or BERT-base, and then normalized according to the statistical features of all word vectors in the modality. Image data is normalized using pixel values, mapping the range of pixel values ​​to the interval [0,1] or [-1,1]. The audio data is normalized by scaling based on the maximum and minimum amplitude of the audio signal. Time series data are normalized using Z-score, and the standardization formula is uniformly expressed as: in, Indicates the first The mean of the original data for a modality is obtained by calculating the arithmetic mean of all data samples for that modality; Indicates the first The standard deviation of the original data for a modality reflects the degree of dispersion of the data for that modality. It is obtained by calculating the square of the deviations of all data samples from the mean and the square root of the mean. For the first The output data of each modality, after standardization, has a mean of 0 and a variance of 1, and can be directly used as input for subsequent models. The preprocessed data is divided into training, validation and test sets in a ratio of 7:2:1. The partitioning process uses stratified sampling to ensure that the modality distribution and privacy attribute distribution of each dataset are consistent with the original data.

[0026] This design clearly defines the multimodal raw data as encompassing various types such as text and images, and provides specific source examples. Different standardization methods are applied to each modality of data to eliminate differences in units and value ranges. Finally, the dataset is divided proportionally, covering a variety of common modalities, making the method widely applicable. Different standardization processes are tailored to the characteristics of each modality of data, ensuring that the data are at a uniform scale before fusion, thus improving the fusion effect. Stratified sampling of the dataset ensures that the distribution of each dataset is consistent with the original data, making model training, validation, and testing more scientific and laying a good foundation for subsequent privacy protection and fusion.

[0027] In one embodiment, constructing a privacy-semantic coupling model includes: Construct privacy feature spaces separately and semantic feature space ; Privacy Feature Space From the set of sensitive attributes generate, The number of sensitive attributes is defined. Sensitive attributes include personal identity information, privacy behavior data, and sensitive preference features. Each sensitive attribute corresponds to an independent privacy dimension, which together constitute a high-dimensional privacy feature space. Semantic feature space It consists of the core semantic representations of each modality. The semantic features of the text modality reflect information such as lexical semantics, syntactic structure, and text theme. The semantic features of the image modality include content such as object category, scene information, and visual attributes. The semantic features of the audio modality cover representations such as speech content, emotional tendency, and audio events. The semantic features of the temporal modality reflect information such as trend changes, periodic features, and key event nodes. Establish a coupling function between the privacy feature space and the semantic feature space. ,in To achieve a bidirectional mapping of the semantic feature space after privacy interference removal, a deep neural network is used. The deep neural network employs an encoder-decoder architecture. The encoder consists of 3-6 Transformer layers, each containing a multi-head attention mechanism, a feedforward neural network, and layer normalization. The decoder uses 2-4 Transformer layers. This architecture not only transmits interference information from the privacy feature space to the semantic feature space for interference localization but also adjusts the strength of privacy interference cancellation based on the constraints of the semantic features. The formula is expressed as: in, This is the parameter set of the coupled model, including learnable parameters such as the weights and biases of the neural network. The initial values ​​of the neural network weights are initialized using a Xavier normal distribution, the weight update rule is based on stochastic gradient descent, and the learning rate is set to a dynamically adjusted mode. The initial learning rate value ranges from [value missing]. After every 50-200 preset iterations, the learning rate is decayed by a decay factor of 0.5-0.8 until it falls below the preset minimum value of 1e-6 and then the decay stops. Privacy variables As an implicit interference factor, it is embedded in the feature interaction process of multimodal fusion; privacy variables are generated through the implicit encoding of sensitive attributes. Specifically, a series of processes such as feature extraction, dimension mapping, and anonymization transformation are performed on sensitive attributes to ensure that they are not directly associated with the original sensitive data, but only retain feature information related to privacy interference, so as to avoid the leakage of the original sensitive data during the model operation. The specific steps of model training include: Initialize the coupled model parameters Set the training batch size to 16-64 and the maximum number of training rounds to 200-500. Input the standardized data from the training set into the model to extract privacy features. and semantic features Generate through coupling function ; Calculate semantic fidelity loss: Combined with location loss due to privacy interference: The inverse mapping of the coupling function yields the training loss of the coupled model: in, To determine the loss weights, the range of values ​​is: ; Update model parameters using backpropagation algorithm Calculate the validation loss on the validation set every 10 training rounds. If the loss does not decrease after 15 consecutive rounds of validation, an early stopping strategy is adopted to stop training and save the current optimal model parameters. After training, the coupling mapping accuracy of the model is verified through the test set, requiring that the semantic feature reconstruction accuracy is not less than 90% and the privacy interference localization accuracy is not less than 85%.

[0028] This design constructs separate privacy and semantic feature spaces, clearly defining their composition. A coupling function is established to achieve bidirectional mapping. A specific neural network architecture is adopted, embedding privacy variables as implicit interference factors into the fusion process. The model training steps are explained in detail, clearly delineating the privacy and semantic spaces for targeted processing. The coupling function and neural network architecture can accurately achieve bidirectional mapping and interference adjustment. The implicit embedding of privacy variables avoids the leakage of original sensitive data. The detailed training steps and parameter settings help the model converge quickly, improve the accuracy of coupling mapping, and balance privacy protection and semantic preservation.

[0029] In one embodiment, based on a privacy-semantic coupling model, privacy noise in multimodal data is automatically corrected and canceled through a regression or mapping model, including: Standardized data for each modality The initial semantic features are extracted using a modality-specific feature extraction network. and privacy interference features ; The initial semantic features focus on the core information carried by the modal data. The text modal feature extraction network uses the encoding layer of the BERT model, the image modal uses the intermediate layer output of ResNet50 or Vision Transformer, the audio modal uses Mel spectral features + CNN network, and the temporal modal uses LSTM network. The privacy interference features are implicitly extracted from the modal data through a specially designed privacy detection network. This privacy detection network is a 2-3 layer fully connected network + Sigmoid activation function. It is trained on a large amount of multimodal data labeled with privacy interference and can accurately identify and separate interference components related to sensitive attributes. Constructing a regression-based privacy-disrupting model This model takes initial semantic features and privacy-intrusion features as input, employs a 2-4 layer fully connected neural network with 256-1024 neurons per layer, and uses ReLU as the activation function. It learns the mapping relationship between privacy noise and semantic features through deep learning, thereby achieving accurate cancellation of privacy noise. Noise correction is achieved by minimizing the privacy-intrusion loss, with the loss function defined as: in, The parameters for the regression model include connection weights of network layers and activation function parameters. The connection weights are initialized using a He normal distribution, and the weight updates employ an adaptive moment estimation optimization algorithm. The first-order momentum decay coefficient is set to 0.9, the second-order momentum decay coefficient is set to 0.999, and the weight decay regularization coefficient ranges from [value missing]. This is used to prevent the model from overfitting; For the first The semantic features of each modality are processed to remove privacy interference. These features retain the original core semantics while eliminating privacy-related interference to the greatest extent possible. The specific steps for training a regression model include: Initialize regression model parameters Set the batch size to 32-128 and the maximum number of training rounds to 150-300. Modal features of the training set Input model, output privacy-free features ; Calculate privacy interference loss Combined with L1 regularization term , The regularization coefficient has a range of values. The total training loss is obtained as follows: Update parameters using backpropagation The privacy-de-suppression effect is evaluated on the validation set every 20 rounds, with privacy leakage risk value and semantic similarity as evaluation metrics. The privacy leakage risk value ranges from [value range missing]. The lower the better, the semantic similarity value range is... The higher the better, to preserve the optimal model; Utilizing the coupling constraint relationship between different modes The constraint relationship is constructed based on the semantic consistency between modalities. That is, when different modal data describe the same target or event, their semantic features after privacy interference removal should maintain logical consistency and information complementarity. This constraint relationship ensures the synergy of privacy noise cancellation during multimodal fusion, avoids semantic distortion caused by privacy removal processing of a single modality, and prevents the cross-transmission of privacy interference between different modalities.

[0030] This design extracts initial semantic and privacy-related interference features from each modality's data, constructs a regression model to counteract privacy noise, details the model structure, loss function, and training steps, and utilizes intermodal coupling constraints to ensure synergy. A dedicated feature extraction network accurately acquires key features from each modality, and the regression model learns the mapping relationship between privacy noise and semantic features, achieving precise cancellation. Detailed training steps and parameter optimization methods prevent model overfitting, and intermodal coupling constraints avoid semantic distortion and cross-transmission of privacy interference caused by single-modal processing, thus improving the quality of multimodal fusion.

[0031] In one embodiment, constraint relationship The specific form is: in, This represents cosine similarity calculation, used to quantify the semantic overlap and consistency between two modalities of privacy-de-identified semantic features, with a value range of [value range missing]. The closer the value is to 1, the higher the semantic consistency. The preset semantic consistency threshold is determined by the following rules: First, calculate the mean cosine similarity of semantic features of different modalities for all samples in the training set without privacy interference. and standard deviation Then, adjust the coefficients according to the semantic fidelity requirements of the application scenario. The range of values ​​is ,final ; When the application scenario has extremely high requirements for semantic consistency, such as medical data fusion, Take a value of 0.9-0.95; When the scenario allows for a certain degree of semantic similarity, such as entertainment content recommendation, Take a value of 0.7-0.85; This constraint ensures that the privacy-de-identified semantic features of different modalities maintain semantic alignment when the calculated features are obtained. When the value is not 0, the model will adjust the parameters of the regression model and the coupling model through backpropagation until the similarity of the semantic features of different modalities meets the preset threshold requirements, so as to ensure that the data after multimodal fusion is semantically coherent and the information is accurate. The steps for incorporating constraints into the training are as follows: During the regression model training process, the constraint loss is calculated every 5 rounds. ; Incorporate the constraint loss into the total loss of the regression model, and update it to... ,in To constrain the loss weights, the range of values ​​is: ; Based on the backpropagation of the updated total loss, the parameters of the regression model and the coupled model are adjusted synchronously to achieve collaborative optimization.

[0032] This design provides specific formulas for constraint relationships, explains the rules for determining semantic consistency thresholds, and outlines the steps for integrating constraint relationships into regression model training and the method for updating the loss function. The specific formulas and threshold determination rules make constraint relationships quantifiable and operable, allowing for flexible adjustments based on different scenarios to meet diverse needs. By integrating them into the training steps and the updated loss function, regression and coupling model parameters can be adjusted simultaneously to achieve collaborative optimization. This ensures semantic alignment of privacy-de-sense features from different modalities, improving the semantic coherence and information accuracy of the fused data.

[0033] In one embodiment, without explicitly exposing the original sensitive data, the true semantic information of multimodal data is recovered through model decoupling and semantic reconstruction, including: The semantic features are separated from the privacy-interference features through the decoupling mechanism of the privacy-semantic coupling model; the decoupling process is achieved through adversarial training to construct a privacy discriminator. and semantic discriminator Among them, privacy discriminator It consists of a 3-layer fully connected network plus a Softmax output layer, with 128-512 hidden neurons, used to distinguish whether the input features contain privacy interference components; semantic discriminator. A lightweight model consistent with the modality feature extraction network structure is used to determine whether the features retain the core semantic information of the original modality; During training, the privacy-semantic coupling model forms an adversarial relationship with the two discriminators. The coupling model is continuously optimized to generate features that can deceive the privacy discriminator so that it cannot identify privacy interference and are recognized by the semantic discriminator to prove the integrity of semantic information. The total loss function for adversarial training is: in, and These are weighting coefficients used to balance the importance of different loss terms. Their determination rules are as follows: Initial settings , The model is trained with fixed weights for the first 100 rounds; after 100 rounds, a privacy protection effectiveness evaluation index is calculated based on the validation set. and semantic fidelity evaluation metrics Privacy protection effectiveness evaluation indicators Use the probability of privacy leakage, with a value range The closer the value is to 1, the better the privacy protection effect. (Semantic fidelity evaluation metric) The mean semantic similarity is used, and the range of values ​​is [not specified]. The closer the value is to 1, the more complete the semantic preservation. Then, the formula is used: The weight coefficients are dynamically adjusted and updated every 50 rounds to ensure that the model achieves the optimal balance between privacy protection and semantic fidelity. To mitigate privacy risks, a binary classification cross-entropy loss is employed, with the following formula: in, Features that are input to the discriminator The value is 1 for true labels, 0 otherwise for privacy interference. Minimizing this loss makes it difficult to detect privacy interference in the features generated by the coupled model. To combat semantic fidelity loss, MSE loss is adopted. ,in The original semantic features reflect the verification results of the semantic discriminator on the core semantics. By minimizing this loss, we ensure that the features after removing privacy interference still retain complete semantic information. The specific steps of adversarial training include: Initialize privacy discriminator and semantic discriminator The parameters are set to 100-200 rounds of adversarial training, and the discriminator learning rate is... ; With fixed parameters for the coupling model and the regression model, train two discriminators: inputting features with privacy interference and features without privacy interference. ,optimization Input the original semantic features and the privacy-de-identified semantic features. ,optimization ; With discriminator parameters fixed, train the coupled model and the regression model: based on the total loss. Backpropagation, update and ; Repeat the above steps, training the discriminator and the generator model alternately. Evaluate the adversarial effect on the validation set every 30 rounds. Stop adversarial training when the privacy discriminator accuracy is close to 50% (random guessing level) and the MSE loss of the semantic discriminator is less than 0.05. Based on the semantic features of each modality after removing privacy interference Feature fusion is performed through a multimodal fusion network. The fusion network adopts a cross-modal attention mechanism, and the rule for determining the attention weights is: calculate the relevance score between the semantic features of each modality and the fusion target. Attention weights are obtained by calculating dot product similarity and then normalizing using the Softmax function. This enables adaptive weighted fusion of semantic features from different modalities; Generate a unified semantic representation Subsequently, privacy-deprecated data for each modality is reconstructed using modality-specific reconstruction networks. Text reconstruction employs a Transformer decoder and vocabulary mapping, image reconstruction uses a convolutional transpose network, audio reconstruction uses LSTM and inverse spectral transform, and time-series data reconstruction uses a fully connected network. Finally, the privacy-preserved multimodal fusion data is obtained by integrating these networks. This data not only removes privacy interference but also fully preserves the core semantics and practical value of the original data; Reconstruct network training parameters: Batch size is 16-64, and learning rate is... The optimizer used is AdamW, with 80-150 training epochs. The loss function is the reconstruction MSE loss, and the reconstruction error on the validation set is required to be less than 0.1.

[0034] This design achieves decoupling through adversarial training, constructing privacy and semantic discriminators. It provides the total loss function and formulas for each loss term in adversarial training, details the training steps, and finally performs multimodal fusion and data reconstruction. The adversarial training mechanism enables the model to achieve an optimal balance between privacy protection and semantic fidelity. Detailed training steps and dynamic weight adjustments ensure stable model training. The multimodal fusion network adaptively weights and fuses features, while the modality-specific reconstruction network accurately restores privacy-deprived data. Ultimately, it yields fused data that protects privacy while preserving core semantics, enhancing the practical value of the data.

[0035] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0036] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A privacy protection method in multimodal data fusion, characterized in that, Includes the following steps: S1. Obtain multimodal raw data of at least two modal types, perform standardization preprocessing on each modal data to eliminate differences in the units and value ranges of different modal data, and obtain standardized multimodal input data; S2. Construct a privacy-semantic coupling model, which includes a privacy feature space and a semantic feature space, establish a bidirectional mapping mechanism between the two, and embed privacy variables as implicit interference factors into the multimodal fusion process; S3. Based on the privacy-semantic coupling model, the privacy noise in multimodal data is automatically corrected and canceled through the data processing model, and mutual constraints are achieved by utilizing the coupling relationship between different modalities. S4. Without explicitly exposing the original sensitive data, restore the true semantic information of multimodal data through model decoupling and semantic reconstruction, and complete the multimodal data fusion under privacy protection.

2. The privacy protection method in multimodal data fusion according to claim 1, characterized in that, The multimodal raw data includes: The data includes at least two of the following: text, image, audio, and time-series data. The standardization preprocessing includes using corresponding normalization methods for different modalities: word vector normalization for text data, pixel value normalization for image data, amplitude normalization for audio data, and Z-score normalization for time-series data. The preprocessed data is divided into training, validation and test sets according to a preset ratio. The division process ensures that the modality distribution and privacy attribute distribution of each dataset are consistent with the original data.

3. The privacy protection method in multimodal data fusion according to claim 1, characterized in that, The steps to build a privacy-semantic coupling model include: A privacy feature space and a semantic feature space are constructed respectively. The privacy feature space is generated by a set of sensitive attributes. The sensitive attributes include at least one of personal identity information, privacy behavior data, and sensitive preference features. Each sensitive attribute corresponds to an independent privacy dimension. The semantic feature space is composed of the core semantic representations of each modality, and the semantic features of different modalities respectively reflect the core information of the corresponding modality; A coupling function between the privacy feature space and the semantic feature space is established, and a bidirectional mapping is achieved through a deep neural network. Privacy variables are embedded as implicit interference factors into the feature interaction process of multimodal fusion.

4. The privacy protection method in multimodal data fusion according to claim 3, characterized in that: The deep neural network adopts an encoder-decoder architecture. The encoder contains 3-6 Transformer encoder layers, each of which includes a multi-head attention mechanism, a feedforward neural network, and layer normalization. The decoder adopts 2-4 Transformer decoder layers. The initial values ​​of the neural network weights are initialized using a Xavier normal distribution, the weight update rule is based on stochastic gradient descent, and the learning rate is set to a dynamic adjustment mode.

5. The privacy protection method in multimodal data fusion according to claim 1, characterized in that, The steps for automatic correction and cancellation of privacy noise based on the privacy-semantic coupling model include: For each modality of standardized data, initial semantic features and privacy interference features are extracted using a feature extraction network; A privacy noise removal model is constructed. Taking the initial semantic features and privacy noise features as input, a deep learning network is used to learn the mapping relationship between privacy noise and semantic features, so as to achieve accurate cancellation of privacy noise. By leveraging the coupling constraints between different modalities, we can ensure the synergy of privacy noise cancellation during multimodal fusion and avoid semantic distortion caused by privacy de-processing of a single modality.

6. The privacy protection method in multimodal data fusion according to claim 5, characterized in that, The feature extraction network includes: Modality-specific feature extraction network and privacy detection network. The modality-specific feature extraction network adopts the corresponding network structure for different modalities. The text modality adopts the encoding layer of the BERT model, the image modality adopts the intermediate layer output of ResNet50 or VisionTransformer, the audio modality adopts Mel spectral features + CNN network, and the temporal modality adopts LSTM network. The privacy detection network is a 2-3 layer fully connected network with a Sigmoid activation function, which can accurately identify and separate interference components related to sensitive attributes.

7. The privacy protection method in multimodal data fusion according to claim 5, characterized in that: The coupling constraint relationship is constructed based on the semantic consistency between modalities, and is specifically quantified by calculating the similarity between the semantic features of different modalities after privacy interference is removed; A preset semantic consistency threshold is set. When the similarity does not meet the threshold requirement, the relevant model parameters are adjusted through backpropagation to ensure that the privacy-de-identified semantic features of different modalities maintain semantic alignment.

8. The privacy protection method in multimodal data fusion according to claim 1, characterized in that, The steps to recover true semantic information through model decoupling and semantic reconstruction include: By using the decoupling mechanism of the privacy-semantic coupling model, semantic features and privacy interference features are separated. The decoupling process is achieved through adversarial training, and a privacy discriminator and a semantic discriminator are constructed. The privacy-semantic coupling model forms an adversarial relationship with the two discriminators. Based on the semantic features of each modality after privacy interference removal, feature fusion is performed through a multimodal fusion network, and a cross-modal attention mechanism is adopted to achieve adaptive weighted fusion of semantic features of different modalities; After generating a unified semantic representation, privacy-de-identified data for each modality is reconstructed using a modality-specific reconstruction network, and then integrated to obtain privacy-preserving multimodal fusion data.

9. The privacy protection method in multimodal data fusion according to claim 8, characterized in that: The privacy discriminator is a 3-layer fully connected network with a Softmax output layer and 128-512 hidden layer neurons. It is used to distinguish whether the input features contain privacy interference components. The semantic discriminator uses a lightweight model consistent with the modality feature extraction network structure to determine whether the features retain the core semantic information of the original modality. During adversarial training, the discriminator and the generator model are trained alternately until the preset training stopping condition is met.

10. The privacy protection method in multimodal data fusion according to claim 8, characterized in that, The attention weights of the multimodal fusion network are determined in the following way: The relevance score between each modality semantic feature and the fusion target is calculated, and the attention weight is obtained by dot product similarity calculation and then normalized by the Softmax function. The modality-specific reconstruction network employs corresponding reconstruction methods for different modalities: text reconstruction uses a Transformer decoder + vocabulary mapping, image reconstruction uses a convolutional transpose network, audio reconstruction uses LSTM + inverse spectral transform, and time-series data reconstruction uses a fully connected network.

Images