Multimodal training data de-sensitization method

Abstract

The application discloses a multi-modal training data desensitization method, which firstly performs structural preprocessing on original multi-modal data, then generates text-weighted image features, image-weighted text features and key cross-modal attention maps through cross-modal feature extraction and collaborative attention fusion, and based on the attention maps and a privacy policy library, innovatively combines a knowledge graph to perform entity identity prior detection, and uses Bayesian inference to adaptively and collaboratively predict sensitivity. Finally, a refined desensitization plan is generated through variational optimization to realize accurate desensitization of the original data. In this way, the inconsistency and "one-size-fits-all" defects of the prior art in multi-modal data privacy protection are solved, the internal value and training availability of the data are maximized while the sensitive information is fully protected, and high-quality, compliant and valuable training data sets are provided for artificial intelligence big models.

Description

Technical Field

[0001] This application relates to the field of intelligent management, and more specifically, to a method for de-identifying multimodal training data. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the rise of multimodal large-scale models, enterprises and research institutions are increasingly collecting and utilizing multimodal data such as images and text. This massive amount of data often contains sensitive personal information or trade secrets. How to maximize the value of data while ensuring data privacy and security, and ensuring data traceability, has become a critical issue that urgently needs to be addressed. Therefore, building an efficient and reliable multimodal training data anonymization solution is particularly important.

[0003] However, existing privacy protection schemes for multimodal data often adopt a patchwork approach, developing separate desensitization tools for each modality. This fragmented processing leads to information silos and serious inconsistencies in desensitization of cross-modal related information. Specifically, when processing multimodal data such as text, images, and video subtitles, it's possible that one modality has been desensitized while another retains sensitive information. Attackers can exploit this inconsistency for identity re-identification, thus posing a risk of cross-modal information leakage. A deeper flaw lies in the "one-size-fits-all" approach and context-blindness of specific elements in the core processing logic of existing mechanisms. It often equates strong correlations between cross-modal features directly with the sensitivity requiring desensitization, ignoring the public attributes of entities and the true intent behind information dissemination. This results in unnecessary desensitization operations when processing information about public figures or authorized public disclosures, incorrectly classifying public information as a privacy risk and severely undermining the intrinsic value and usability of training data. Furthermore, treating sensitivity assessment as a static attribute determined by entity category makes the mechanism unable to dynamically adapt to complex contextual changes and unable to distinguish the sensitivity differences of the same entity under different disclosure intentions, resulting in unintended harm and overprotection of data value.

[0004] Therefore, an optimized method for desensitizing multimodal training data is needed. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a method for desensitizing multimodal training data. First, the original multimodal data undergoes structured preprocessing. Then, through cross-modal feature extraction and collaborative attention fusion, text-weighted image features, image-weighted text features, and a key cross-modal attention map are generated. Based on this attention map and a privacy policy library, a knowledge graph is innovatively combined to perform entity identity prior detection, and Bayesian inference is used for adaptive collaborative prediction of sensitivity. Finally, variational optimization generates a refined desensitization plan, achieving accurate desensitization of the original data. This approach solves the inconsistencies and "one-size-fits-all" shortcomings of existing technologies in multimodal data privacy protection, ensuring that sensitive information is fully protected while maximizing the preservation of the data's intrinsic value and training usability, providing high-quality, compliant, and valuable training datasets for large-scale artificial intelligence models.

[0006] According to one aspect of this application, a method for de-identifying multimodal training data is provided, comprising:

[0007] Acquire raw multimodal data;

[0008] The original multimodal data is preprocessed in a structured manner to obtain image patch sequences and text token sequences;

[0009] Cross-modal feature extraction and collaborative attention fusion are performed on image patch sequences and text token sequences to obtain image patch feature sequences weighted by text information, text token feature sequences weighted by image information, and cross-modal attention maps;

[0010] Based on cross-modal attention maps and a privacy policy library, sensitivity collaborative prediction and desensitization strategy generation are performed on the image block feature sequence after text information weighting and the text token feature sequence after image information weighting to obtain a desensitization plan;

[0011] The original multimodal data is anonymized based on the anonymization plan to obtain anonymized multimodal data.

[0012] Compared with existing technologies, this application provides a method for desensitizing multimodal training data. First, it performs structured preprocessing on the original multimodal data. Then, through cross-modal feature extraction and collaborative attention fusion, it generates text-weighted image features, image-weighted text features, and a key cross-modal attention map. Based on this attention map and a privacy policy library, it innovatively combines knowledge graphs for entity identity prior detection and utilizes Bayesian inference for adaptive collaborative prediction of sensitivity. Finally, variational optimization generates a refined desensitization plan, achieving accurate desensitization of the original data. This approach addresses the inconsistencies and "one-size-fits-all" shortcomings of existing technologies in multimodal data privacy protection, ensuring that sensitive information is fully protected while maximizing the preservation of the data's intrinsic value and training usability. This provides high-quality, compliant, and valuable training datasets for large-scale artificial intelligence models. Attached Figure Description

[0013] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0014] Figure 1 This is a flowchart of a multimodal training data desensitization method according to an embodiment of this application;

[0015] Figure 2 This is a schematic diagram of the data flow in the multimodal training data desensitization method according to an embodiment of this application. Detailed Implementation

[0016] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0017] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0018] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0019] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0020] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0021] The technical solution of this application proposes a method for desensitizing multimodal training data. Figure 1 This is a flowchart of a multimodal training data desensitization method according to an embodiment of this application. Figure 2 This is a system architecture diagram of a multimodal training data desensitization method according to an embodiment of this application. Figure 1 and Figure 2 As shown, the multimodal training data desensitization method according to an embodiment of this application includes the following steps: S1, acquiring original multimodal data; S2, performing structured preprocessing on the original multimodal data to obtain image patch sequences and text token sequences; S3, performing cross-modal feature extraction and collaborative attention fusion on the image patch sequences and text token sequences to obtain text-weighted image patch feature sequences, image-weighted text token feature sequences, and cross-modal attention maps; S4, based on the cross-modal attention maps and a privacy policy library, performing sensitivity collaborative prediction and desensitization strategy generation on the text-weighted image patch feature sequences and image-weighted text token feature sequences to obtain a desensitization plan; S5, performing desensitization processing on the original multimodal data based on the desensitization plan to obtain desensitized multimodal data.

[0022] Specifically, S1 involves acquiring raw multimodal data. Raw multimodal data refers to an initial dataset containing at least two different information representations that has not undergone any feature extraction, anonymization, or advanced semantic understanding processing. These representations, or modalities, can be visual information (such as images and video clips), textual information (such as text descriptions and comments), or any other data type that can collaboratively express semantics. In practice, this typically involves retrieving data from diverse data sources, which may include internal data warehouses, public datasets, data streams captured in real-time by various sensors, or information such as images, audio, and video obtained through web crawling.

[0023] Specifically, step S2 involves performing structured preprocessing on the raw multimodal data to obtain image patch sequences and text token sequences. It should be understood that raw multimodal data, such as images and text, typically have diverse formats, sizes, and representations; for example, images may vary in resolution, and text may vary in length and contain unstructured words. This unstructured nature makes them unsuitable as direct input to multimodal models. Therefore, in the technical solution of this application, structured preprocessing is performed on the raw multimodal data to transform the heterogeneous and irregular raw data into a unified and standardized format, providing standardized input to the model and ensuring smooth collaborative processing and feature extraction between different modalities, thereby effectively supporting subsequent sensitivity prediction and desensitization strategy generation.

[0024] In practice, firstly, the image data from the original multimodal data is input into an image processor for size standardization and image patch segmentation to obtain the image patch sequence. Size standardization involves adjusting all input images to a preset uniform size (e.g., 224x224 pixels). This can be achieved through scaling, cropping, or padding techniques to eliminate the impact of image size differences on model training and ensure that all images have the same spatial dimension during feature extraction. Next, image patch segmentation decomposes the standardized image into a series of fixed-size image patches, which may or may not overlap. For example, a 224x224 pixel image can be segmented into several 16x16 pixel image patches. These patches are arranged from left to right and top to bottom, forming a sequence of visual features, similar to a word sequence in text. This allows the image data to be processed by models such as Transformer in a serialized form.

[0025] Next, the text input from the original multimodal data is processed by a text processor for word segmentation and word embedding to obtain the text token sequence. Word segmentation involves dividing a continuous text string into the smallest units with independent semantics. The segmentation rules differ for different languages; for example, English is typically segmented by spaces and punctuation marks, while Chinese requires more complex lexical analysis to identify word boundaries. This step transforms the original text into discrete word or character sequences. Subsequently, word embedding is performed on the segmented text from the original multimodal data. Word embedding maps these discrete text tokens to a high-dimensional, continuous vector space to capture the semantic and syntactic relationships between words, ensuring that semantically similar words are close in distance in the vector space. This transforms non-numerical text information into numerical feature vectors that the model can understand and compute. Commonly used word embedding methods include Word2Vec, GloVe, or context-sensitive embeddings generated based on pre-trained language models (such as BERT). These embedding vectors collectively constitute the text token sequence, providing a foundation for subsequent text feature extraction and cross-modal fusion.

[0026] Specifically, in step S3, cross-modal feature extraction and collaborative attention fusion are performed on the image patch sequence and text token sequence to obtain a text-weighted image patch feature sequence, a text-weighted text token feature sequence, and a cross-modal attention map. It should be understood that although the original image patch sequence and text token sequence have undergone preliminary structuring processing, they are still independent representations, lacking cross-modal context awareness. To identify potential sensitive information in multimodal data, especially those sensitive information that is cross-modal related, such as faces in images and names in text, it is necessary to deeply fuse information from different modalities, enabling the model to understand how image content is supplemented by text descriptions, and vice versa. This fusion can reveal the strong correlation between image regions and text words, not only solving the problem of information silos between modalities in traditional methods, but also forming the basis for accurate sensitivity prediction and the generation of effective desensitization strategies.

[0027] In practice, firstly, the image patch sequence and the text token sequence are input into the image encoder and text encoder respectively to obtain the image embedding sequence and text embedding sequence. Although the image patch sequence and text token sequence already possess a certain structure, they are still low-level features and have not yet fully captured the complex visual semantics of the image and the deep contextual information of the text. In order to achieve subsequent cross-modal interaction and sensitivity analysis, these sequences must be abstracted and feature extracted at a higher level, elevating the pixel-level image patches and word-level text tokens into semantically rich embedding vectors. These embedding vectors not only encode the information of each individual element, but also provide high-quality input for the subsequent cross-modal attention mechanism, enabling the model to accurately identify the potential correlations and sensitivities between different modalities.

[0028] Specifically, this process comprises two parallel sub-steps designed to encode deep features for the image and text, respectively. First, the system inputs a sequence of image patches, obtained after size normalization and patch segmentation, into an image encoder. The image encoder is typically a deep neural network model, such as the VisionTransformer (ViT) based on the Transformer architecture, or classic convolutional neural networks (CNNs) like ResNet and Inception. For each image patch, the image encoder performs a series of complex feature extraction operations, including but not limited to convolution, pooling, non-linear transformations of activation functions, and self-attention mechanisms. These operations aim to learn and extract visual features from local image patches, such as edges, textures, shapes, and higher-level semantic concepts. Finally, each image patch is mapped to a fixed-dimensional continuous vector, which together constitute the image embedding sequence. Each image embedding vector contains rich semantic information and contextual features of its corresponding image patch, enabling the model to understand image content in an abstract and efficient manner.

[0029] Simultaneously, the system inputs the text token sequence obtained after word segmentation and word embedding into the text encoder. The text encoder is typically a pre-trained language model based on the Transformer architecture (such as BERT, RoBERTa, etc.) or a recurrent neural network (RNN) such as LSTM, GRU, etc. These encoders can process sequential data and capture long-range dependencies and contextual semantics in the text. For each word embedding in the text token sequence (i.e., the word vector obtained in the preprocessing stage), the text encoder transforms and refines it through a multi-layered self-attention mechanism and a feedforward network. In this process, the embedding vector of each text token not only contains its own semantic information but also incorporates its contextual information within the entire text sequence, such as its relationship with preceding and following words and its grammatical role in the sentence. Ultimately, each text token is transformed into a high-dimensional, context-aware continuous vector, and these vectors together constitute the text embedding sequence. These text embedding sequences are more semantically rich and contextualized than the original word embeddings, providing a deeper level of text representation for subsequent cross-modal fusion.

[0030] Next, using the image embedding sequence as the query vector and the text embedding as the key and value vectors, it is input into the cross-modal attention module to obtain a text-weighted image patch feature sequence and an attention weight matrix as the cross-modal attention map. Although the previous step has obtained the image embedding sequence and the text embedding sequence, they are still independent modal representations. In order to accurately identify sensitive information associated with cross-modal relationships, such as the correspondence between a face in an image and a name in text, the technical solution of this application generates a more context-aware image feature representation by weighting the text information onto the image patch features. This allows subsequent sensitivity prediction to more accurately determine whether a certain region in the image (such as a face) is sensitive due to a strong association with sensitive words (such as a name) in the text. At the same time, the attention weight matrix generated in this process, as the cross-modal attention map, intuitively reveals the specific association strength and pattern between the image and the text, providing key guidance for sensitivity collaborative prediction.

[0031] Specifically, firstly, the system uses the image embedding sequence obtained in the previous step as the query vector. This means that the embedding vector of each image patch will actively search for the relevant parts in the text sequence. At the same time, the text embedding sequence is used as the key vector and value vector. Specifically, as the key, each vector in the text embedding sequence represents the content information of a certain token in the text, which is used to match the image query vector. As the value, each vector in the text embedding sequence contains the actual semantic content of the token, which is used to contribute to the weighted output after a successful match.

[0032] Next, attention weights are calculated: these query vectors (image embeddings) and key vectors (text embeddings) are input into the cross-modal attention module. Inside this module, the association strength between each image query vector and all text key vectors is first evaluated by calculating the similarity between them. This similarity is typically achieved through a dot product operation, i.e., matrix multiplication of the image query vector and the text key vector. Then, to prevent the dot product from becoming too large and causing gradient instability, these similarity scores are divided by the square root of the key vector dimension (i.e., scaling). Finally, a Softmax function is applied to the scaled similarity scores to normalize the result to a probability distribution between 0 and 1, thus obtaining the attention weights. These attention weights form a matrix where each element (i, j) represents the association strength between the i-th image patch and the j-th text token.

[0033] Finally, the resulting attention weight matrix is ​​applied to the text value vectors. Specifically, each image query vector is summed in a weighted manner based on its attention weight with all text key vectors. This means that for a given image patch, if it is highly correlated with a word in the text, then the value vector of that word will have a larger weight in the weighted sum. In this way, the original features of each image patch are fused with contextual information from the text modality, resulting in a text-weighted image patch feature sequence. This weighted sequence contains both the visual information of the image and the associated semantic information of the text, providing a foundation for subsequent, more refined sensitivity analysis.

[0034] Furthermore, using the text embedding sequence as the query vector and the image embedding as the key and value vectors, this sequence is input into the cross-modal attention module to obtain a text token feature sequence weighted by image information. To achieve comprehensive perception and accurate identification of sensitive information in multimodal data, it is necessary not only to integrate image features into the textual context but also for text features to perceive and utilize the visual information provided by the image. This weighting of text features by image information enhances the text's ability to refer to visual content and its contextual understanding, especially for identifying entities mentioned in the text but visually sensitive. In this way, a more visually perceptive text feature representation can be generated, providing a more comprehensive and accurate input for subsequent sensitivity collaborative prediction and desensitization strategy generation based on cross-modal attention maps.

[0035] Specifically, firstly, the system uses the text embedding sequence obtained in the previous step as the query vector. This means that the embedding vector of each token in the text sequence will actively search for related visual information in the image sequence. At the same time, the image embedding sequence is used as the key vector and value vector. Specifically, as the key, each vector in the image embedding sequence represents the visual content information of a certain block in the image, which is used to match the text query vector. As the value, each vector in the image embedding sequence contains the actual visual semantic content of the image block, which is used to contribute to the weighted output after a successful match.

[0036] Next, these query vectors (text embeddings) and key vectors (image embeddings) are input into the cross-modal attention module. Inside this module, the association strength between each text query vector and all image key vectors is first evaluated by calculating the similarity between them. This similarity is typically achieved through a dot product operation, i.e., matrix multiplication of the text query vector and the image key vector. Then, to prevent the dot product from becoming too large and causing gradient instability, these similarity scores are divided by the square root of the key vector dimension (i.e., scaling). Finally, a Softmax function is applied to the scaled similarity scores to normalize the result to a probability distribution between 0 and 1, thus obtaining the attention weights. These attention weights form a matrix, where each element (i, j) represents the association strength between the i-th text token and the j-th image patch.

[0037] Finally, the resulting attention weight matrix is ​​applied to the image value vectors. Specifically, each text query vector is weighted and summed in relation to the image value vectors based on its attention weight with all image key vectors. This means that for a given text token, if it is highly correlated with a region in the image, the value vector of that region will have a larger weight in the weighted sum. In this way, the original features of each text token are fused with contextual information from the image modality, resulting in a weighted text token feature sequence. This weighted sequence contains the semantic information of the text as well as its associated visual information, enhancing the text's ability to perceive and express the image content, and providing a foundation for subsequent, more refined sensitivity analysis.

[0038] Specifically, in step S4, based on cross-modal attention maps and a privacy policy library, sensitivity collaborative prediction and desensitization strategy generation are performed on the image patch feature sequence after weighting text information and the text token feature sequence after weighting image information to obtain a desensitization plan. It should be understood that the core technical flaw of the original mechanism lies in its one-size-fits-all processing logic and its blindness to the context of specific elements. This mechanism directly equates the strong correlation between cross-modal features with the sensitivity requiring desensitization. This simplistic assumption ignores a crucial specific element: the public attributes of the entity and the intent of information dissemination. This flaw leads to unnecessary desensitization operations when processing information about public figures or authorized public disclosures due to the strong correlation between images and text descriptions. This not only incorrectly classifies public information as a privacy risk but also substantially undermines the intrinsic value and usability of the training data, especially in tasks requiring the model to learn public figure recognition or specific public scenario knowledge. Furthermore, assessing sensitivity as a static attribute determined by entity category makes the mechanism unable to dynamically adapt to complex contextual changes and unable to distinguish the sensitivity differences of the same entity under different disclosure intents, thus causing unintended harm and overprotection of data value.

[0039] To address the aforementioned shortcomings, this application proposes an adaptive sensitivity assessment and policy generation method based on Bayesian inference and variational optimization. This method transforms the desensitization process from a static classification mapping problem into a dynamic, adaptive risk quantification and policy optimization process.

[0040] In practice, firstly, based on the cross-modal attention map, entity pair sets are extracted from the text-weighted image patch feature sequence and the image-weighted text token feature sequence. It should be understood that although the previous step has generated context-rich weighted feature sequences and intuitive cross-modal attention maps, these are still low-level representations. To achieve higher-level semantic understanding, especially for entity identification using knowledge graphs (such as determining whether an entity is a public figure), it is necessary to clarify which image regions and which text words jointly refer to a single real-world entity. In the technical solution of this application, the extraction of entity pair sets from the text-weighted image patch feature sequence and the image-weighted text token feature sequence aims to identify these semantically strongly correlated entity combinations from complex feature sequences, providing structured input for subsequent knowledge graph queries and prior detection, thereby overcoming the shortcomings of traditional methods such as unclear entity boundaries and ineffective context awareness.

[0041] In this process, the system first extracts sets of matching text-weighted image block features and image-weighted text token features from the text-weighted image block feature sequence and the image-weighted text token feature sequence based on a comparison between the cross-modal attention map and a preset threshold. By comparing the attention weights in the attention map with a preset threshold, feature pairs with association strength exceeding the threshold can be identified. Next, the system calculates the positional mean vector between each matching text-weighted image block feature and image-weighted text token feature in the set of matching text-weighted image block features and image-weighted text token features to obtain the joint entity representation vector of each entity pair. The calculated joint entity representation vectors together constitute the entity pair set, providing a high-level semantic input for subsequent knowledge graph-based prior evidence acquisition and sensitivity assessment.

[0042] Next, the joint entity representation vectors of each entity pair in the entity pair set are input into the knowledge graph embedding model to obtain a prior evidence set. It should be understood that traditional sensitivity assessment mechanisms often rely solely on the feature correlations within the data, failing to effectively integrate common sense or facts from the external world. This leads to "contextual blindness" and over-sensitization problems in sensitivity judgment. To assign a precise contextual identity to each entity, distinguish whether a strongly related entity is personal privacy or public information, thereby avoiding unnecessary desensitization operations and preserving the intrinsic value of the data, the technical solution of this application associates entities identified within the data with an external knowledge base to obtain prior knowledge about entity attributes. In this process, the knowledge graph embedding model attempts to link entities to a large external knowledge base, such as a list of public figures or a geographic location information database, thereby obtaining a key prior evidence set for each entity pair, i.e., the probability that it possesses public attributes. This prior evidence quantifies the contextual information of the entity identity, providing a crucial basis for subsequent risk assessment and solving the deficiency of the original mechanism in lacking external world knowledge to distinguish between private and public attributes of entities.

[0043] Subsequently, the image patch feature sequence weighted by text information and the text token feature sequence weighted by image information are input into the image sensitivity prediction head and the text sensitivity prediction head to obtain an initial sensitivity prediction set. Although in previous steps, image and text features have been mutually weighted to obtain rich contextual information, enabling features of each modality to perceive the content of the other modality, this perception is merely a feature-level fusion and does not directly determine whether these fused features represent sensitive information. This step aims to transform these context-enhanced features into explicit sensitivity predictions, forming a preliminary identification of potentially sensitive content in multimodal data. This serves as the basis for subsequent, more accurate evaluation considering external knowledge (such as public attributes), thereby avoiding over-desensitization decisions before understanding the entity's public attributes. The resulting initial sensitivity prediction set represents the model's preliminary judgment of potentially sensitive information in multimodal data, i.e., the prior sensitivity probability.

[0044] In this process, firstly, the image patch feature sequence weighted by text information is input into the image sensitivity prediction head. The image sensitivity prediction head is a trained deep learning classifier whose task is to analyze the features of each image patch incorporating text context and predict the probability that it contains sensitive information. For example, this prediction head is a feedforward neural network that receives the feature vector of the image patch as input, transforms it through multiple hidden layers and activation functions, and finally outputs a probability value between 0 and 1, representing the probability that the image patch contains sensitive visual information (such as faces, license plates, ID cards, etc.). This process is performed independently for each image patch in the sequence, thus generating a sensitivity prediction sequence corresponding to the length of the image patch sequence.

[0045] Simultaneously, the image-weighted text token feature sequence is input into a text sensitivity prediction head. This head, also a trained deep learning classifier, analyzes the features of each text token incorporating image context and predicts the likelihood that it contains sensitive information. For example, this prediction head can be a feedforward neural network or a Transformer-based classification layer. It receives the feature vector of the text token as input and outputs a probability value between 0 and 1, representing the probability that the text token contains sensitive text information (such as names, addresses, phone numbers, or specific trade secrets). This process is performed independently for each text token in the sequence, thus generating a sensitivity prediction sequence corresponding to the length of the text token sequence.

[0046] Finally, the sensitivity prediction sequences generated by the image sensitivity prediction head and the text sensitivity prediction head are integrated to obtain an initial sensitivity prediction set. This set contains the preliminary sensitivity probability or classification result for each image patch and each text token in the multimodal data. It represents the model's judgment of sensitivity based on the patterns it has learned before introducing external knowledge.

[0047] Then, a posterior sensitivity assessment based on Bayesian inference is performed on the initial sensitivity prediction set and the prior evidence set to obtain a posterior sensitivity score set. Considering that static sensitivity classification cannot accurately reflect the true risk of an entity when it possesses public attributes, a more rigorous probabilistic method must be adopted to revise the initial judgment using newly acquired evidence. Specifically, a posterior sensitivity assessment based on Bayesian inference is performed on the initial sensitivity prediction set and the prior evidence set. This process applies Bayes' theorem, using the prior evidence of public attributes obtained in the previous step as a condition to update the initial sensitivity probability given by the model prediction head. Specifically, the posterior sensitivity assessment based on Bayesian inference is performed on the initial sensitivity prediction set and the prior evidence set using the following formula:

[0048]

[0049]

[0050] In this formula, This represents the posterior sensitivity score, i.e., the score obtained after observing evidence that an entity possesses public attributes. Afterwards, the entity is a sensitive event. The probability of; It is the model's initial sensitivity prediction, i.e., the prior probability; It is the likelihood function, representing the probability that a truly sensitive entity will be detected as having public attributes; this value is usually small. This represents the probability that a non-sensitive entity is detected as having public attributes. Thus, a more accurate posterior sensitivity score, adjusted for contextual information, is calculated. For entities with strong public attributes, even if their initial sensitivity prediction is high, their final posterior sensitivity score will be significantly and reasonably lowered, thereby mathematically achieving intelligent exemptions for public figures and authorized public information.

[0051] Finally, a de-identification plan is obtained by generating a variational optimization-based de-identification strategy for the posterior sensitivity score set based on a privacy policy library. It should be understood that a simple "category-operation" mapping strategy is too rigid and cannot make a fine trade-off between privacy protection and data utility. Therefore, in the technical solution of this application, the de-identification decision is constructed as an optimization problem, with the goal of solving for an optimal de-identification operation for each entity to minimize an overall loss functional that includes residual privacy risks and data utility loss. For discrete operation options, this problem simplifies to solving for the following optimal operation... :

[0052]

[0053] In this formula, It is an entity The selected optimal desensitization operation; It is the posterior sensitivity score calculated for the entity in the previous step; It is the set of all optional desensitization operations; It is an operation The privacy protection effectiveness is rated between 0 and 1; It is to perform operations The resulting loss of data utility; This is an adjustable hyperparameter used to balance the weights between privacy and utility. In this way, going beyond simple rule matching, the optimal response strategy for each entity is dynamically and mathematically solved under the current risk level and system policy preferences. The result is the generation of a refined de-identification plan, where each de-identification instruction is the optimal solution derived through an optimization algorithm based on specific risk quantification, thus ensuring the adaptability, rationality, and interpretability of the entire de-identification process.

[0054] In this preferred technical solution, the system no longer indiscriminately anonymizes all strongly related entities. Instead, it intelligently identifies and differentiates between public and private entities. For the former, the system significantly reduces their risk rating to preserve their information integrity; for the latter, it matches the optimal anonymization intensity based on their precisely quantified risk level. This transforms the anonymization process from extensive coverage to precise control, greatly reducing unnecessary damage to data value and improving the quality and usability of the final multimodal training dataset.

[0055] Specifically, step S5 involves desensitizing the original multimodal data based on a desensitization plan to obtain desensitized multimodal data. In other words, the desensitization process translates the results of all the aforementioned complex analysis, prediction, and strategy generation steps into concrete operations on the original data. This step is crucial for fulfilling the privacy commitment, transforming the theoretical desensitization plan into a practically applicable dataset. Specifically, this process involves parsing each desensitization instruction in the desensitization plan, identifying the specific sensitive entity (whether a region in an image or a word in text) it refers to in the original multimodal data, and then transforming the original data accordingly based on the desensitization operation type and intensity specified in the instruction. For example, if the desensitization plan instructs for moderate blurring of a specific facial region in an image, the system will precisely locate the pixel coordinates of that facial region and apply an appropriate image processing algorithm (such as Gaussian blur) to blur the facial features while still preserving some overall contour information. If the instruction is to anonymize a name entity in the text, the system will find the name in the original text and replace it with a generic placeholder (e.g., "[Name]") or a new code generated by the system. Throughout the process, the system iterates through all instructions in the anonymization plan, ensuring that every identified sensitive entity is properly handled according to the predetermined strategy. This approach avoids indiscriminate "one-size-fits-all" anonymization of all strongly related entities, instead intelligently identifying and differentiating between public and private entities. For public entities, the system significantly reduces their risk rating to preserve their information integrity; while for private entities, the optimal anonymization strength is matched based on their precisely quantified risk level. Finally, the original data after all anonymization operations forms an anonymized multimodal data set.

[0056] In this process, desensitization is the actual modification, blurring, anonymization, or removal of sensitive information in the original multimodal data according to the specific instructions in the desensitization plan. The desensitized multimodal data is the final dataset obtained after desensitization, in which all identified sensitive information has been protected according to the predetermined strategy, while preserving the non-sensitive parts of the data and the overall usability to the maximum extent, greatly reducing unnecessary damage to the value of the data, and improving the quality and usability of the final output multimodal training dataset.

[0057] Taking the scheme in this application as an example, suppose an original multimodal dataset contains an image of a residential community and its accompanying detailed address text. If the desensitization plan includes an instruction to completely cover the house numbers appearing in the image, while generalizing the specific street names and building numbers in the text (e.g., changing "XX Street XX Number" to "XX Street XX Community"), then during this step, the system will first locate the pixel region of the house number in the original image and replace it with a solid-color square to achieve complete coverage. Subsequently, the system will parse the text data, find the specific street name and building number information, and modify it into a broader regional description. Finally, the resulting desensitized multimodal data will contain an image with the house numbers covered and a text with generalized address information. This data protects the privacy of residents while retaining the general information of the community's location, and can still be used for tasks such as geographic area analysis or community landscape recognition.

[0058] In summary, the multimodal training data desensitization method according to the embodiments of this application is explained. First, the original multimodal data undergoes structured preprocessing. Then, through cross-modal feature extraction and collaborative attention fusion, text-weighted image features, image-weighted text features, and a key cross-modal attention map are generated. Based on this attention map and a privacy policy library, a knowledge graph is innovatively combined to perform entity identity prior detection, and Bayesian inference is used for adaptive collaborative prediction of sensitivity. Finally, variational optimization generates a refined desensitization plan, achieving accurate desensitization of the original data. This approach solves the inconsistencies and "one-size-fits-all" shortcomings of existing technologies in multimodal data privacy protection, ensuring that sensitive information is fully protected while maximizing the preservation of the data's intrinsic value and training usability, providing high-quality, compliant, and valuable training datasets for large-scale artificial intelligence models.

[0059] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for desensitizing multimodal training data, characterized in that, include: Acquire raw multimodal data; The original multimodal data is preprocessed in a structured manner to obtain image patch sequences and text token sequences; Cross-modal feature extraction and collaborative attention fusion are performed on image patch sequences and text token sequences to obtain image patch feature sequences weighted by text information, text token feature sequences weighted by image information, and cross-modal attention maps; Based on cross-modal attention maps and a privacy policy library, sensitivity collaborative prediction and desensitization strategy generation are performed on the image block feature sequence after text information weighting and the text token feature sequence after image information weighting to obtain a desensitization plan; The original multimodal data is desensitized based on the desensitization plan to obtain desensitized multimodal data; Cross-modal feature extraction and collaborative attention fusion are performed on image patch sequences and text token sequences to obtain text-weighted image patch feature sequences, image-weighted text token feature sequences, and cross-modal attention maps, including: The image patch sequence is input into the image encoder and the text token sequence is input into the text encoder to obtain the image embedding sequence and the text embedding sequence; Using the image embedding sequence as the query vector and the text embedding as the key vector and value vector, the cross-modal attention module is input to obtain the image patch feature sequence and attention weight matrix after text information weighting, which are used as the cross-modal attention map; Using the text embedding sequence as the query vector and the image embedding as the key and value vectors, we input them into the cross-modal attention module to obtain the image information weighted text token feature sequence. Based on cross-modal attention maps and a privacy policy library, sensitivity collaborative prediction and desensitization strategy generation are performed on the text-weighted image patch feature sequence and the image-weighted text token feature sequence to obtain a desensitization plan, including: Based on cross-modal attention maps, entity pair sets are extracted from image patch feature sequences weighted by text information and text token feature sequences weighted by image information; The joint entity representation vector of each entity pair in the entity pair set is input into the knowledge graph embedding model to obtain the prior validation data set; The image block feature sequence after weighting the text information and the text token feature sequence after weighting the image information are input into the image sensitivity prediction head and the text sensitivity prediction head to obtain the initial sensitivity prediction set; A posterior sensitivity assessment based on Bayesian inference is performed on the initial sensitivity prediction set and the prior evidence set to obtain a posterior sensitivity score set. Based on a privacy policy library, a variational optimization-based desensitization strategy is generated from the posterior sensitivity score set to obtain a desensitization plan.

2. The multimodal training data anonymization method according to claim 1, characterized in that, The raw multimodal data undergoes structured preprocessing to obtain image patch sequences and text token sequences, including: The image data from the original multimodal data is input into an image processor for size normalization and image block segmentation to obtain the image block sequence; The text input from the original multimodal data is processed by a text processor for word segmentation and word embedding to obtain the text token sequence.

3. The multimodal training data desensitization method according to claim 2, characterized in that, Based on cross-modal attention maps, entity pair sets are extracted from image patch feature sequences weighted by text information and image token feature sequences weighted by text information, including: Based on the comparison between cross-modal attention maps and preset thresholds, a set of matching text-weighted image block features and text-weighted text token features is extracted from the text-weighted image block feature sequence and the image-weighted text token feature sequence. Calculate the positional mean vector between each set of matched text-weighted image patch features and image-weighted text token features to obtain the joint entity representation vector for each entity pair.

4. The multimodal training data desensitization method according to claim 3, characterized in that, To obtain a posterior sensitivity score set, a Bayesian inference-based posterior sensitivity evaluation is performed on the initial sensitivity prediction set and the prior evidence set. This includes: performing a Bayesian inference-based posterior sensitivity evaluation on the initial sensitivity prediction set and the prior evidence set using the following formula: in, For initial sensitivity prediction, This represents the probability that a truly sensitive entity is detected as having public properties. This represents the probability that a non-sensitive entity is detected as having public attributes.

5. The multimodal training data anonymization method according to claim 1, characterized in that, The de-identification plan is obtained by generating a variational optimization-based de-identification strategy from the posterior sensitivity score set based on a privacy policy library. This includes: generating a variational optimization-based de-identification strategy from the posterior sensitivity score set based on the privacy policy library using the following formula: in, It is an entity The selected optimal desensitization operation; It is the posterior sensitivity score calculated for the entity in the previous step; It is the set of all optional desensitization operations; It is an operation The privacy protection effectiveness is rated between 0 and 1; It is to perform operations The resulting loss of data utility; It is an adjustable hyperparameter used to balance the weight between privacy and utility.

Images