A medical multi-modal image data privacy de-identification processing method

Abstract

This invention discloses a method for privacy de-identification processing of medical multimodal image data. The invention relates to the field of intelligent medical information processing and privacy protection technology, and includes the following steps: Step S1: Acquire multimodal raw data, which includes medical image sequences and corresponding text records; Step S2: Perform a first round of multimodal joint sensitive information detection, detecting explicit and implicit identifier regions and features in each medical image. This method for privacy de-identification processing of medical multimodal image data verifies the thoroughness of de-identification from both active attack and passive analysis perspectives through a resilient attack network and a feature residue detection network in a bimodal adversarial verification process. The joint determination by both ensures that the pass of either one is insufficient to constitute a final conclusion on privacy security, thus forming a dual guarantee.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical information processing and privacy protection technology, specifically a method for privacy de-identification processing of medical multimodal image data. Background Technology

[0002] With the rapid development of intelligent healthcare and remote diagnostic systems, medical imaging data is increasingly used in clinical decision support, disease screening, and medical research. Multimodal medical data, in particular, such as endoscopic images, CT images, MRI images, and corresponding diagnostic text records, has become a crucial input for intelligent assisted diagnostic systems. To improve diagnostic efficiency, more and more medical institutions are uploading this multimodal imaging data to cloud servers for centralized processing and analysis. However, this data processing method also brings serious privacy risks. Multimodal medical data contains highly sensitive information such as patient identity, medical history, and imaging characteristics. If this information is obtained by malicious attackers during transmission or storage, it will pose a serious threat to patient privacy.

[0003] Currently, several methods have attempted to protect the privacy of medical data through encryption technology. For example, patent document CN118260798A describes a "Privacy Protection Method for Multimodal Medical Data Based on Deep Learning." This method encrypts multimodal medical data using the BFV fully homomorphic encryption algorithm and constructs a deep learning model suitable for encrypted environments, achieving anomaly identification and classification in the encrypted state. This method protects the privacy and security of data during cloud processing to a certain extent and has high theoretical value and practical significance.

[0004] However, this method still has the following shortcomings: First, its encryption and model training processes are highly coupled, resulting in high system complexity and making it difficult to deploy quickly in actual medical systems; second, this method mainly focuses on the model's reasoning ability in the encrypted state, and does not perform special de-identification processing for explicit identity information in image data, such as patient name, ID, and device information, which poses a potential privacy leakage path; third, the method still relies on a relatively complex neural network structure for the fusion processing of multimodal data, resulting in large computational overhead and making it difficult to run efficiently on resource-constrained edge devices.

[0005] To address these issues, this invention proposes a privacy-preserving de-identification method for medical multimodal image data. This method aims to solve problems in existing technologies, such as incomplete privacy protection, lack of de-identification, and high computational complexity during the transmission and processing of multimodal medical image data. By automatically detecting and de-identifying the identity information in the image data, combined with a lightweight feature extraction and fusion mechanism, an efficient, secure, and deployable privacy protection scheme for multimodal image data is achieved. Summary of the Invention

[0006] The purpose of this invention is to provide a method for privacy de-identification of medical multimodal image data to solve the problems mentioned in the background art.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a method for privacy de-identification processing of medical multimodal image data, comprising the following steps:

[0008] Step S1: Obtain multimodal raw data, which includes medical image sequences and corresponding text records;

[0009] Step S2: Perform the first round of multimodal joint sensitive information detection, detect explicit and implicit marker regions in each medical image, and output the set of sensitive regions and the comprehensive sensitivity of each sensitive region;

[0010] Step S3: Perform adaptive de-identification processing on the sensitive regions based on the comprehensive sensitivity of each sensitive region to obtain the first round of de-identified image;

[0011] Step S4: Perform bimodal adversarial verification on the first round of de-identified images. The bimodal adversarial verification includes: using a regenerative attack network to attempt to recover sensitive information from the first round of de-identified images and calculating the recovery success rate, and using a feature residual detection network to calculate the probability of residual sensitive features. When the recovery success rate is not greater than a first threshold and the probability of residual sensitive features is not greater than a second threshold, the verification is deemed to have passed; otherwise, the verification is deemed to have failed and the coordinates of the failed area are recorded.

[0012] Step S5: Using the de-identified images that fail verification and their failure region coordinates as feedback signals, update the detection network parameters in step S2 and the de-identification parameters in step S3 through meta-learning.

[0013] Step S6: Repeat steps S2 to S5 at least once until the pass rate of two consecutive rounds of verification is not less than the third threshold and the fluctuation range of the pass rate is not greater than the fourth threshold. Stop the iteration and output the final de-identified image sequence.

[0014] Preferably, the multimodal joint sensitive information detection in step S2 includes:

[0015] Level 1 detection: A lightweight texture-text separation network is used to quickly scan medical images and output a binary mask of candidate text regions;

[0016] Second-level detection: After cropping the region indicated by the binary mask, the data is input into an attention-enhanced character recognition network, which outputs candidate sensitive entities, their confidence scores, and bounding boxes.

[0017] Level 3 detection: Perform cross-modal semantic alignment between the candidate sensitive entities and the entities extracted by named entity recognition in the text records, mark only the successfully aligned entities as sensitive information, and calculate the explicit labeling sensitivity based on the alignment similarity;

[0018] Simultaneously, medical images are input into an implicit identifier feature extraction network, which outputs an implicit sensitivity score.

[0019] The explicit sensitivity score and the implicit sensitivity score are nonlinearly fused to obtain a comprehensive sensitivity score.

[0020] Preferably, the cross-modal semantic alignment uses a BERT-based semantic similarity matrix to calculate the similarity between each candidate sensitive entity and each named entity in the text record. When the maximum similarity is greater than the fifth threshold, the alignment is considered successful, and the candidate sensitive entity is marked as sensitive information.

[0021] Preferably, the adaptive de-identification process in step S3 includes:

[0022] When the overall sensitivity of the sensitive area is greater than the sixth threshold, the pixel value is transformed by a reversible polynomial perturbation. The reversible polynomial perturbation is to perform a third-order polynomial operation on the pixel value in the encrypted domain, and the transformation coefficient is determined by the overall sensitivity.

[0023] When the overall sensitivity is greater than the seventh threshold but not greater than the sixth threshold, adaptive Gaussian blur is used, and the size of the blur kernel is positively correlated with the overall sensitivity.

[0024] When the overall sensitivity is no greater than the seventh threshold, only pixel-level Gaussian noise is injected.

[0025] Preferably, the restorative attack network is a generator in a generative adversarial network, whose input is a de-identified image and output is a restored image; the restoration success rate is calculated as the ratio of the number of regions in the restored image and the original image whose structural similarity index at the coordinates of the failed region is greater than an eighth threshold to the total number of sensitive regions.

[0026] Preferably, the feature residual detection network is a binary classification convolutional neural network, whose input is the difference image between the de-identified image and the original image, and whose output is the probability value of residual sensitive features.

[0027] Preferably, the meta-learning method in step S5 adopts a model-independent meta-learning framework, which uses the weighted sum of the recovery success rate of the failed samples and the probability of residual sensitive features in the previous iteration as the total loss, calculates the gradients for the detection network parameters and the de-identification parameters respectively, and updates the gradients in one or more steps; the detection network parameters include the weights of the texture-text separation network, the weights of the attention-enhanced character recognition network, and the weights of the implicit identifier feature extraction network; the de-identification parameters include the polynomial coefficients in the invertible polynomial perturbation, the kernel size of the adaptive Gaussian blur, and the variance of the pixel-level Gaussian noise.

[0028] Preferably, the feedback signal in step S5 further includes a structural difference map of the recovered image and the original image in the failure region, which verifies the recovered image of the failed sample output by the restorative attack network. The structural difference map is used to guide the detection network to improve the detection sensitivity of the corresponding position of the failure region and its neighboring regions in the next iteration.

[0029] Preferably, the stopping iteration condition in step S6 is as follows: the pass rate of two consecutive rounds of verification is not less than the ninth threshold, and the absolute value of the difference between the pass rate of the next round and the pass rate of the previous round is not greater than the tenth threshold; wherein the pass rate is equal to the ratio of the number of medical images that have passed verification to the total number of medical images processed in the current round.

[0030] Preferably, after step S6, the method further includes:

[0031] Step S7: Use a lightweight convolutional neural network to extract image feature maps from the final de-identified image sequence, and use a lightweight pre-trained language model to extract text semantic vectors from the text records;

[0032] Step S8: Generate a random orthogonal matrix, multiply the image feature map by the random orthogonal matrix to obtain the perturbed image features, wherein the random orthogonal matrix satisfies that its transpose multiplied by itself is an identity matrix;

[0033] Step S9: A gated attention fusion layer is used to fuse the perturbed image features with the text semantic vector to obtain a fused feature vector;

[0034] Step S10: The final de-identified image sequence is transmitted using stream encryption, the fused feature vector is transmitted using homomorphic encryption, and the text record is transmitted using attribute encryption.

[0035] This invention provides a method for privacy de-identification processing of medical multimodal image data. It has the following beneficial effects:

[0036] This method constructs a closed-loop architecture combining multimodal joint sensitive information detection and bimodal adversarial verification, which can simultaneously eliminate traceable features such as explicit text identifiers and implicit device fingerprints in images. The three-level heterogeneous detection combined with cross-modal semantic alignment ensures that the identification of sensitive information does not rely on single-modal cues, effectively avoiding the missed detection problems that occur when relying solely on images or text.

[0037] In the bimodal adversarial verification, the restorative attack network and the feature residue detection network verify the thoroughness of de-identification from the perspectives of active attack and passive analysis, respectively. Their joint judgment ensures that the success of either one is insufficient to represent a final conclusion on privacy and security, thus forming a dual guarantee. A meta-learning-driven cyclical self-feedback mechanism feeds back information of failed samples to the detection network and de-identification parameters, enabling the system to specifically strengthen its sensitivity and processing intensity for weak areas in each iteration. After a limited number of iterations, a stable and reliable de-identification effect can be achieved.

[0038] Furthermore, this method ensures strong privacy protection while also considering the feasibility of practical deployment and the integrity of diagnostic functions. A lightweight feature extraction network and a reversible orthogonal perturbation mechanism ensure that key anatomical features in images are fully preserved after de-identification, and the perturbed features can be reversibly recovered when the key is held, balancing privacy protection and diagnostic traceability. A gated attention fusion layer achieves adaptive fusion of image features and textual semantics, obtaining effective joint feature representations without the need for complex neural networks. A hierarchical transmission strategy employing stream encryption, homomorphic encryption, and attribute encryption is used for different data types, satisfying the needs of encrypted inference in the cloud while implementing role-based access control. The overall solution allows preprocessing and iterative verification to be completed on edge devices, reducing reliance on cloud computing. Simultaneously, a closed-loop feedback mechanism ensures the consistency of multiple processing results, providing a repeatable and verifiable technical path for privacy protection of multimodal medical data. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the overall method of the medical multimodal image data privacy de-identification processing method of the present invention;

[0040] Figure 2 This is a three-level flowchart of the multimodal joint sensitive information detection process of a medical multimodal image data privacy de-identification processing method of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Please see Figure 1 and Figure 2 This invention provides a technical solution: a method for privacy de-identification processing of medical multimodal image data, comprising the following steps:

[0043] Step S1: Obtain multimodal raw data, which includes medical image sequences and corresponding text records;

[0044] Step S2: Perform the first round of multimodal joint sensitive information detection, detect explicit and implicit marker regions in each medical image, and output the set of sensitive regions and the comprehensive sensitivity of each sensitive region;

[0045] Step S3: Perform adaptive de-identification processing on the sensitive areas based on the comprehensive sensitivity of each sensitive area to obtain the first round of de-identified image;

[0046] Step S4: Perform bimodal adversarial verification on the first round of de-identified images. The bimodal adversarial verification includes: using a recovery attack network to attempt to recover sensitive information from the first round of de-identified images and calculating the recovery success rate, and using a feature residue detection network to calculate the probability of residual sensitive features. When the recovery success rate is not greater than the first threshold and the probability of residual sensitive features is not greater than the second threshold, the verification is deemed to have passed; otherwise, the verification is deemed to have failed and the coordinates of the failed area are recorded.

[0047] Step S5: Using the de-identified images that fail verification and their failure region coordinates as feedback signals, update the detection network parameters in step S2 and the de-identification parameters in step S3 through meta-learning.

[0048] Step S6: Repeat steps S2 to S5 at least once until the pass rate of two consecutive rounds of verification is not less than the third threshold and the fluctuation range of the pass rate is not greater than the fourth threshold. Stop the iteration and output the final de-identified image sequence.

[0049] It should be further explained that in step S1, the system first acquires multimodal raw data, which includes medical image sequences generated by endoscope, CT or MRI equipment and corresponding structured or unstructured text records, wherein the text records contain basic patient information, diagnostic reports and equipment metadata.

[0050] Step S2 performs the first round of multimodal joint sensitive information detection: For each medical image, it is copied into two copies. The first copy is input into a lightweight texture-text separation network, which uses a four-layer depthwise separable convolution to output a binary mask to mark the regions that may contain text or watermarks. After cropping and scaling the regions indicated by the mask to a specified size, it is input into a convolutional recurrent neural network with a self-attention mechanism to output candidate sensitive entities and their confidence scores and bounding boxes. At the same time, the text records are extracted using named entity recognition to extract the patient's name, ID number, examination number, equipment model, and date entity set.

[0051] For each candidate sensitive entity, a BERT-based semantic similarity matrix is ​​used to calculate its cosine similarity with each entity in the text record. When the maximum similarity is greater than 0.85, the alignment is considered successful, and the candidate sensitive entity is marked as explicit sensitive information. Explicit label sensitivity is then calculated based on the alignment similarity and confidence score. On the other hand, a second copy of the medical image is input into a pre-trained implicit label feature extraction network. This network uses a knowledge-distilled and compressed ResNet-18 structure and outputs an implicit sensitivity score between 0 and 1, used to characterize traceable implicit labels such as device fingerprints, scanning patterns, or pixel distribution features present in the image.

[0052] The explicit sensitivity score and the implicit sensitivity score are fused using a non-linear formula:

[0053]

[0054] in, Overall sensitivity, with a value ranging from 0 to 1; : Explicitly identify sensitivity, determined by text alignment confidence; Implicit sensitivity score, output by the feature extraction network;

[0055] The overall sensitivity φ is calculated, and the region is included in the sensitive region set when φ is greater than 0.6.

[0056] Step S3 performs adaptive de-identification processing based on the overall sensitivity of each sensitive region: if the overall sensitivity is greater than 0.9, a reversible polynomial perturbation is used, that is, a third-order polynomial transformation is performed on the pixel value in the homomorphic encryption domain:

[0057]

[0058] in, Original pixel value, normalized to 0-1; : Encrypted pixel value; a0, a1, a2, a3: Polynomial coefficients, obtained by linear mapping of comprehensive sensitivity, and this transformation can be completely reversed when the key is held; Homomorphic encryption modulus;

[0059] If the overall sensitivity is greater than 0.6 and not greater than 0.9, adaptive Gaussian blur is used, with the blur kernel size as follows:

[0060]

[0061] in, The standard deviation of the Gaussian blur kernel is used to calculate the kernel size; The baseline standard deviation is 2.0. Overall sensitivity;

[0062] If the overall sensitivity is no greater than 0.6, only Gaussian noise with a variance of 0.01 is injected. The first round of de-identified image is obtained after processing.

[0063] Step S4 performs bimodal adversarial verification on the first round of de-identified images. The first channel uses a restorative attack network, which is a U-Net structure generator with skip connections. It is pre-trained with a large number of artificially synthesized de-identified-original image pairs, and the loss function is a weighted sum of perceptual loss and adversarial loss. The first round of de-identified images are input into this network, and the restored images are output. The structural similarity index of the local regions within the sensitive region bounding boxes of the restored images and the original images is calculated. The ratio of the number of regions with a structural similarity index greater than 0.3 to the total number of sensitive regions is used as the restoration success rate.

[0064] The second channel employs a residual feature detection network, which is a binary classification convolutional neural network. The input is the difference image between the first-round de-identified image and the original image, and the output is a residual sensitive feature probability value between 0 and 1. A first threshold of 0.05 and a second threshold of 0.1 are set. Verification is considered successful when the recovery success rate is no greater than 0.05 and the residual sensitive feature probability is no greater than 0.1; otherwise, verification is considered unsuccessful, and the coordinates of the failed region and the structural difference map between the recovered image output by the recovery attack network and the original image in the failed region are recorded.

[0065] Step S5 uses the de-identified images that failed verification, along with their failure region coordinates and structural difference maps, as feedback signals to update the detection network parameters from Step S2 and the de-identification parameters from Step S3 using a meta-learning approach. Specifically, a model-independent meta-learning framework is employed, where the total loss is calculated by weighting the recovery success rate of all failed verification samples from the previous iteration with the probability of residual sensitive features. Gradients are then calculated for this total loss with respect to the detection network parameters and the de-identification parameters. The detection network parameters include the convolutional weights of the texture-text separation network, the CRNN weights of the attention-enhanced character recognition network, and the ResNet weights of the implicit identifier feature extraction network. The de-identification parameters include the polynomial coefficients of the invertible polynomial perturbation, the kernel size coefficient of the adaptive Gaussian blur, and the variance of the Gaussian noise. A gradient update step is then performed based on the calculated gradients, with a step size of 0.001.

[0066] Meanwhile, the structural difference map is used as additional feedback to guide the detection network to enhance the detection sensitivity of the failed region coordinates and its eight neighboring locations in the next iteration. Specifically, the pixels at these locations are assigned a weight coefficient of 2.0 in the loss function of the texture-text separation network.

[0067] Step S6 repeats steps S2 to S5 at least once. After each iteration, the verification pass rate is calculated, which is the ratio of the number of verified medical images to the total number of medical images processed in the current round. A third threshold is set to 0.98, and a fourth threshold is set to 0.01. Iteration stops when the verification pass rate for two consecutive iterations is not less than 0.98, and the absolute value of the difference between the verification pass rate of the later round and the previous round is not greater than 0.01. If the conditions are not met, the next round continues, with a maximum of three iterations before forced termination. The final output is a de-identified image sequence after all iterations, where each image no longer contains sensitive information that can be identified by the recovery attack network or the feature residual detection network.

[0068] The multimodal joint sensitive information detection in step S2 includes:

[0069] Level 1 detection: A lightweight texture-text separation network is used to quickly scan medical images and output a binary mask of candidate text regions;

[0070] Second-level detection: After cropping the region indicated by the binary mask, the data is input into the attention-enhanced character recognition network, which outputs candidate sensitive entities, their confidence scores, and bounding boxes.

[0071] Level 3 detection: Perform cross-modal semantic alignment between candidate sensitive entities and entities extracted from text records by named entity recognition, mark only successfully aligned entities as sensitive information, and calculate explicit labeling sensitivity based on alignment similarity;

[0072] Simultaneously, medical images are input into an implicit identifier feature extraction network, which outputs an implicit sensitivity score.

[0073] The explicit sensitivity score and the implicit sensitivity score are nonlinearly fused to obtain the comprehensive sensitivity score.

[0074] It should be further explained that in the multimodal joint sensitive information detection in step S2, for each medical image, the first level of detection is first performed: the image is input into a lightweight texture-text separation network, which uses four depthwise separable convolutions. Each convolution is followed by batch normalization and ReLU activation function. The last layer is activated by Sigmoid and outputs a binary mask of the same size as the input image. The pixel position with a value of 1 in the mask indicates that the area may contain text, watermarks or printing marks.

[0075] Then, the second-level detection is performed: all connected regions are extracted based on a binary mask, the bounding rectangle of each connected region is expanded by four pixels and then cropped, and the cropped region is scaled to a size of 128 by 128 pixels. The scaled image is then input into a convolutional recurrent neural network, which consists of two convolutional layers, one recurrent layer, and one transcription layer. The convolutional layers use 3x3 convolutional kernels, the recurrent layer uses a bidirectional long short-term memory network, and the transcription layer uses a connection-temporal classification loss function. The network outputs the character recognition sequence and the corresponding confidence score for each connected region, as well as the bounding box coordinates of the region.

[0076] The third level of detection is then performed: the text record is input into a named entity recognition model, which uses a bidirectional long short-term memory network combined with a conditional random field. Pre-trained annotations include patient name, ID number, examination number, equipment model, and date. The model outputs the category and start and end positions of all entities in the text record. For each candidate sensitive entity output from the second level of detection, its recognized string is extracted. This string and the text content of each named entity in the text record are then input into a BERT-based semantic similarity calculation module. This module first segments and encodes the two strings separately, then calculates the cosine similarity between the two encoded vectors. The maximum value among all calculated results is taken as the alignment similarity of the candidate sensitive entity.

[0077] A similarity threshold is set. When the alignment similarity is greater than this threshold, the alignment is considered successful, and the candidate sensitive entity is marked as sensitive information, with its bounding box and entity category recorded. Simultaneously, the confidence level of the candidate sensitive entity is multiplied by the alignment similarity to obtain the explicit sensitivity level of the sensitive region. For candidate sensitive entities that fail to align, if their confidence level is greater than another confidence threshold, they are marked as suspicious sensitive information and sent to the manual review queue.

[0078] While the above three-level detection is running, another copy of the same medical image is input into the implicit identifier feature extraction network. This network is a ResNet-18 variant that has been knowledge distilled and compressed. Its last fully connected layer outputs a single value between 0 and 1. This value is the implicit sensitivity score, which is used to characterize implicit traceable identifiers such as device sensor noise patterns, pixel distribution statistical characteristics, or scanning grid patterns in the image.

[0079] Finally, the explicit sensitivity score is denoted as w, the implicit sensitivity score is denoted as s, and the comprehensive sensitivity score φ is calculated using the nonlinear fusion formula φ=1-(1-w)×(1-s). If φ is greater than 0.6, the sensitive region and its comprehensive sensitivity score are added to the sensitive region set and used as input for subsequent de-identification processing.

[0080] Cross-modal semantic alignment uses a BERT-based semantic similarity matrix to calculate the similarity between each candidate sensitive entity and each named entity in the text record. When the maximum similarity is greater than the fifth threshold, the alignment is considered successful, and the candidate sensitive entity is marked as sensitive information.

[0081] It should be further explained that in the process of cross-modal semantic alignment, the string identified by the candidate sensitive entity is first recorded as the text to be matched, and the text content of each named entity extracted by named entity recognition in the text record is used as the reference text.

[0082] For each text to be matched and a reference text, encoding is performed using a pre-trained BERT model: the text to be matched and the reference text are each converted into a word sequence by the BERT tokenizer. [CLS] and [SEP] tags are added to the beginning and end of each sequence respectively before being input into the BERT encoder. The vector output corresponding to the [CLS] tag is taken as the semantic encoding vector of that text. The semantic encoding vector of the text to be matched is denoted as... The semantic encoding vector of the reference text is denoted as .

[0083] calculate and The cosine similarity between them is calculated using the following formula:

[0084]

[0085] in, Cosine similarity between two vectors; : BERT semantic encoding vector of candidate sensitive entities; : BERT semantic encoding vector of named entities in text records; · represents vector dot product, ||·|| represents vector magnitude.

[0086] The calculated cosine similarity value is used as the semantic similarity between the text to be matched and the reference text. After traversing all named entities in the text record, a set of semantic similarity values ​​is obtained, and the maximum value among them is taken as the maximum semantic similarity between the candidate sensitive entity and the text record.

[0087] A fifth threshold is set at 0.85. When the maximum semantic similarity is greater than 0.85, the candidate sensitive entity is considered to have successfully aligned with a named entity in the text record, and the candidate sensitive entity is marked as sensitive information. Simultaneously, the category of the aligned named entity is recorded, such as patient name, ID number, examination number, equipment model, or date. If the maximum semantic similarity is not greater than 0.85, the alignment is considered to have failed, and the candidate sensitive entity is not marked as sensitive information. However, if its original confidence score is greater than 0.9, it is sent to the manual review queue for further confirmation.

[0088] In the above process, the BERT model uses a pre-trained, finely tuned version for the Chinese medical field. Its parameters are loaded during system initialization and remain fixed during subsequent meta-learning feedback to ensure the stability of semantic similarity calculation.

[0089] The adaptive de-identification process in step S3 includes:

[0090] When the overall sensitivity of the sensitive area is greater than the sixth threshold, the pixel value is transformed by reversible polynomial perturbation. The reversible polynomial perturbation is to perform a third-order polynomial operation on the pixel value in the encrypted domain, and the transformation coefficient is determined by the overall sensitivity.

[0091] When the overall sensitivity is greater than the seventh threshold but not greater than the sixth threshold, adaptive Gaussian blur is used, and the size of the blur kernel is positively correlated with the overall sensitivity.

[0092] When the overall sensitivity is no greater than the seventh threshold, only pixel-level Gaussian noise is injected.

[0093] It should be further explained that in the adaptive de-identification process of step S3, the comprehensive sensitivity φ of the sensitive region is first obtained, and its value ranges from 0 to 1. The sixth threshold is set to 0.9 and the seventh threshold is set to 0.6.

[0094] When φ is greater than 0.9, a reversible polynomial perturbation is applied to each pixel in the sensitive region for transformation. Specifically, the original pixel value p is divided by 255 and normalized to the interval [0,1] to obtain... The original pixel value p ranges from 0 to 255; then the value of the third-order polynomial is calculated. The polynomial coefficients a0, a1, a2, and a3 are determined by the comprehensive sensitivity φ through a linear mapping, with the mapping relationships being a0 = 0.1 × φ, a1 = 0.5 + 0.2 × φ, a2 = 0.2 × φ, and a3 = 0.1 × φ. Multiply by 255 and round to the nearest integer to obtain the encrypted pixel value p′. This transformation is performed in the homomorphic encryption domain and can reversibly recover the original pixel value when the polynomial coefficients and transformation formula are known.

[0095] When φ is greater than 0.6 and not greater than 0.9, adaptive Gaussian blurring is applied to the sensitive region: first, the size of the blur kernel is calculated. That is, when φ=0.6, σ=2.0, and when φ=0.9, σ=6.0. The value of σ is rounded up to an odd number. Then, a Gaussian kernel matrix is ​​generated with this value as the standard deviation. The blurred pixel value is obtained by weighted averaging of each pixel in the region and its neighborhood.

[0096] When φ is not greater than 0.6, Gaussian noise is injected independently only for each pixel in the sensitive area: a Gaussian random number with a mean of 0 and a variance of 0.01 is generated, the random number is added to the pixel value, and then truncated to the range of 0 to 255.

[0097] All three processing methods described above apply only to the sensitive regions, leaving pixels outside the sensitive regions unchanged to ensure that non-sensitive diagnostic features in the image are not destroyed. After processing, the results of each sensitive region are stitched together with the unprocessed non-sensitive regions to obtain a complete de-identified image.

[0098] The reconstructive attack network is a generator in generative adversarial networks. Its input is a de-identified image, and its output is a reconstructed image. The success rate of reconstruction is calculated as the ratio of the number of regions in the reconstructed image and the original image whose structural similarity index at the coordinates of the failed region is greater than the eighth threshold to the total number of sensitive regions.

[0099] It should be further explained that in the first channel of the bimodal adversarial verification in step S4, the restorative attack network adopts a generator in the generative adversarial network architecture. The specific structure of this generator is a U-Net network with skip connections: the encoder path contains four downsampling blocks, each of which consists of two convolutional layers and one max pooling layer, with the number of feature channels being 64, 128, 256, and 512 respectively; the decoder path contains four upsampling blocks, each of which consists of one transposed convolutional layer and two convolutional layers, with the number of feature channels being 512, 256, 128, and 64 respectively; wherein, the convolutional kernel size of the convolutional layer is 3×3, the stride is 1, and the padding is 1; the size of the max pooling layer is 2×2, and the stride is 2; the convolutional kernel size of the transposed convolutional layer is 2×2, and the stride is 2; the encoder and decoder are connected by skip connections to concatenate the feature maps of the corresponding layers in the channel dimension.

[0100] The generator takes a de-identified image as input (256×256 pixels) and outputs a restored image. The restorative attack network is pre-trained on a training set where each sample contains a pair of images: the original image and its corresponding de-identified image, generated by the adaptive de-identification process in step S3. The loss function during training is a weighted sum of perceptual and adversarial losses. The perceptual loss is calculated using L2 distance extracted from feature maps by a pre-trained VGG16 network, while the adversarial loss is the binary cross-entropy output of the discriminator.

[0101] During the verification phase, the first round of de-identified images is input into the generator, and the restored image is output. For the coordinates of the failed regions determined in step S4, i.e., the bounding boxes of sensitive regions, local image patches corresponding to the bounding boxes are extracted from both the restored image and the original image, and a structural similarity index is calculated for each image patch.

[0102] This index is calculated based on the product of three components: luminance, contrast, and structure. The luminance comparison function is:

[0103]

[0104] The contrast comparison function is:

[0105]

[0106] The structure comparison function is:

[0107]

[0108] in, Brightness comparison function; Contrast comparison function; : Structure comparison function; , : The mean values ​​of the x and y values ​​of the image patch; , Standard deviation of image patches; Covariance; C1, C2, and C3 are stability constants, with C1 = (0.01L) 2 C2 = (0.03L) 2 C3 = C2 / 2, and L is the dynamic range of pixel values, 255.

[0109] The eighth threshold is set to 0.3. The number of local image patches with a structural similarity index greater than 0.3 is counted, and this number is divided by the total number of sensitive regions to obtain the restoration success rate. This restoration success rate is compared with the first threshold of 0.05. If it is greater than 0.05, the verification fails.

[0110] The feature residual detection network is a binary classification convolutional neural network. Its input is the difference image between the de-identified image and the original image, and its output is the probability value of residual sensitive features.

[0111] It should be further explained that in the second channel of the bimodal adversarial verification in step S4, the feature residual detection network adopts a binary classification convolutional neural network structure. The input image of this network is generated as follows: the de-identified image and the original image are subtracted pixel by pixel at the same pixel position to obtain a difference image. The value of each pixel in the difference image is equal to the pixel value of the de-identified image minus the pixel value of the original image. The result may be positive or negative. Then, 128 is added to the difference and truncated to the range of 0 to 255 to convert it into a grayscale image, resulting in a single-channel difference grayscale image. The size of this difference grayscale image is 256×256 pixels.

[0112] The structure of the feature residual detection network is as follows: First convolutional layer, kernel size 5×5, stride 1, padding 2, output channels 32, followed by batch normalization layer and ReLU activation function; First max pooling layer, pooling window 2×2, stride 2; Second convolutional layer, kernel size 3×3, stride 1, padding 1, output channels 64, followed by batch normalization layer and ReLU activation function; Second max pooling layer, pooling window 2×2, stride 2; Third convolutional layer, kernel size... The first layer is a 3×3 convolutional layer with a stride of 1, padding of 1, and 128 output channels, followed by a batch normalization layer and a ReLU activation function; the second layer is a max pooling layer with a pooling window of 2×2 and a stride of 2; the third layer is a convolutional layer with a kernel size of 3×3, a stride of 1, padding of 1, and 256 output channels, followed by a batch normalization layer and a ReLU activation function; the fourth layer is a global average pooling layer that compresses the feature map into a 256-dimensional vector; the fifth layer is a fully connected layer that maps the 256-dimensional vector into a 2-dimensional output, followed by a Softmax activation function.

[0113] The network outputs probability values ​​for two categories: the first category corresponds to "existence of residual sensitive features," and the second category corresponds to "absence of residual sensitive features." The probability value of the first category is used as the probability value of the residual sensitive features, with a value ranging from 0 to 1. The network is pre-trained using a training set. Positive samples in the training set are generated from the original images through weak de-identification processing in step S3, while negative samples are generated from the original images through strong de-identification processing in step S3. During training, a binary cross-entropy loss function is used, the optimizer is Adam, and the learning rate is set to 0.001. Positive samples in the training set contain residual sensitive features, while negative samples do not.

[0114] During the verification phase, the probability value of residual sensitive features output by the network is compared with the second threshold of 0.1. If the probability value is greater than 0.1, the verification is deemed to have failed, indicating that there are still detectable sensitive feature information remaining in the de-identified image.

[0115] The meta-learning method in step S5 adopts a model-independent meta-learning framework. It uses the weighted sum of the recovery success rate of the failed samples and the probability of residual sensitive features in the previous iteration as the total loss. It calculates the gradients for the detection network parameters and the de-identification parameters and updates the gradients in one or more steps. The detection network parameters include the weights of the texture-text separation network, the weights of the attention-enhanced character recognition network, and the weights of the implicit identifier feature extraction network. The de-identification parameters include the polynomial coefficients in the invertible polynomial perturbation, the kernel size of the adaptive Gaussian blur, and the variance of the pixel-level Gaussian noise.

[0116] It should be further explained that in step S5, the meta-learning method adopts a model-independent meta-learning framework, and the specific execution process is as follows: First, collect all samples that failed verification in the previous iteration. For each sample that failed verification, record its recovery success rate. and residual sensitive feature probability The loss value for this sample is obtained by weighting and summing the recovery success rate with the probability of residual sensitive features:

[0117]

[0118] in, The loss value for a single sample that fails verification; Recovery success rate: 0-1%; The probability of residual sensitive features is 0 to 1; α and β are preset weight coefficients, with α set to 0.4 and β set to 0.6.

[0119] The total loss for this round is obtained by summing the losses of all failed samples and dividing by the total number of failed samples. .

[0120] Then, calculate the total loss separately. Regarding the gradients of the detection network parameters and de-identification parameters. The detection network parameters include: the convolutional weights and biases of each layer of the texture-text separation network, which contains four depthwise separable convolutions, each with a kernel size of 3×3 and 16, 32, 64, and 128 channels respectively; the convolutional layer weights and biases of the attention-enhanced character recognition network, the weight matrix and bias terms of the bidirectional long short-term memory network, and the connection temporal classification parameters of the transcription layer; and all convolutional layer weights, batch normalization parameters, and fully connected layer weights of the ResNet-18 compressed version of the implicit label feature extraction network.

[0121] The de-identification parameters include: polynomial coefficients a0, a1, a2, and a3 in the invertible polynomial perturbation, each coefficient being a floating-point number, with its value range determined by the linear mapping of the comprehensive sensitivity; and the kernel size coefficient of the adaptive Gaussian blur, i.e., the formula... In The value is 2.0; the variance of pixel-level Gaussian noise is 0.01.

[0122] Based on the calculated gradient, a gradient update is performed according to the update rules of the model-independent meta-learning framework: for each parameter θ, the update formula is:

[0123]

[0124] in, Updated model parameters; Model parameters before update; Meta-learning rate, set to 0.001; The gradient of the total loss with respect to the parameter θ; : The average loss of all samples that failed the verification;

[0125] The updated detection network parameters and de-identification parameters will be used in steps S2 and S3 of the next iteration. The entire update process is performed once after each iteration, and gradients are calculated only using samples that fail verification; samples that pass verification are not included in the update. Furthermore, to maintain training stability, the L2 norm of the detection network parameters is pruned after each update, limiting the L2 norm to within 10.0. The above meta-learning process is executed for a maximum of three rounds, with the updated parameters from each round serving as the initial parameters for the next round.

[0126] The feedback signal in step S5 also includes a structural difference map of the restored image and the original image in the failure region, which is output by the restorative attack network to verify the failed sample. The structural difference map is used to guide the detection network to improve the detection sensitivity of the corresponding position of the failure region and its neighboring regions in the next iteration.

[0127] It should be further explained that in the feedback signal of step S5, for each sample that fails verification, in addition to recording the coordinates of the failure area, a structural difference map is also generated. The specific generation method of the structural difference map is as follows: Local image patches are extracted from the recovered image output by the recovery attack network and the original image at the same failure area coordinates. The two local image patches are of the same size, both being rectangular areas expanded eight pixels outwards from the failure area bounding box. The structural difference value is calculated pixel-by-pixel for these two local image patches. The formula for calculating the structural difference value is 1 minus the local structural similarity index value at that pixel location. The local structural similarity index value is calculated pixel-by-pixel on the image patch using a sliding window to obtain a similarity index matrix of the same size as the image patch. Then, each element in this matrix is ​​subtracted from 1 to obtain the structural difference matrix. Each element in this matrix ranges from 0 to 1; a larger value indicates a more significant structural difference at that location. The sliding window size is 7×7 with a step size of 1. The structural difference matrix is ​​normalized to the range of 0 to 255 and saved as a grayscale image, which is the structural difference map.

[0128] The structural difference map is input into the training module of the detection network as part of the feedback signal. In the next iteration, when the texture-text separation network in the detection network is trained, higher loss weights are assigned to the locations corresponding to the failed regions and their neighboring regions based on the structural difference map: Specifically, for each sample that fails the verification, the structural difference map is scaled to the same size as the binary mask, and then for each pixel location in the binary mask, its original loss weight is multiplied by (1 + the normalized value of the structural difference map at that pixel location), so that the pixel location with the larger structural difference value receives a larger loss weight in the next round of training, thereby guiding the detection network to pay more attention to these regions; the default value of the original loss weight is 1.

[0129] Simultaneously, for the bounding rectangle of the failed region coordinates, it is extended outward by eight pixels to form an extended region. The loss weights of all pixels within this extended region are additionally multiplied by a fixed coefficient of 1.5. After this weighting, the loss function of the detection network becomes a weighted binary cross-entropy loss, where the weight of each pixel is determined by the product of the base weight, the structural difference map weight, and the extended region weight. In this way, the detection network will produce a stronger detection response in the next iteration for regions where the recovery attack was successful or where feature residues are severe, as well as their surrounding regions, thereby improving its sensitivity to these regions.

[0130] The specific stopping iteration condition in step S6 is: the pass rate of two consecutive rounds of verification is not less than the ninth threshold, and the absolute value of the difference between the pass rate of the next round and the pass rate of the previous round is not greater than the tenth threshold; where the pass rate is equal to the ratio of the number of medical images that have passed verification to the total number of medical images processed in the current round.

[0131] It should be further explained that in the stopping iteration condition judgment in step S6, the total number of medical images processed in the current round is first defined as... This number is equal to the number of images in the medical image sequence obtained in step S1. For each iteration, after completing the bimodal adversarial verification in step S4, the number of medical images that passed the verification is counted. The verification pass rate is defined as follows: the image restoration success rate is no greater than the first threshold and the probability of residual sensitive features is no greater than the second threshold. Therefore, the verification pass rate for this round is:

[0132]

[0133] in, The pass rate for the current round of verification; The number of medical images that have passed verification; : Total number of medical images processed in the current round;

[0134] This ratio is expressed as a decimal and ranges from 0 to 1. The ninth threshold is set to 0.98, and the tenth threshold is set to 0.01.

[0135] Stopping iteration requires two conditions to be met simultaneously: the first condition is the pass rate of the current round of verification. Not less than 0.98, and the pass rate of the previous round of verification. It is also not less than 0.98; the second condition is the absolute value of the difference between the current round's pass rate and the previous round's pass rate. Not greater than 0.01. In the first iteration, there is no data from the previous iteration; therefore, after the first iteration, no stopping condition is determined, and the process directly proceeds to the second iteration.

[0136] Starting from the second iteration, after completing step S4 and calculating the pass rate for each iteration, the pass rate stored in the previous iteration is retrieved, and the two conditions mentioned above are checked. If both conditions are met, the iteration stops, and the subsequent steps S5 and S6 are not repeated. The de-identified image sequence obtained in the current iteration is directly output as the final result. If either condition is not met, the repeated operations in steps S5 and S6 continue, and the next iteration begins. Furthermore, to prevent infinite loops, a maximum of three iterations is preset. When the third iteration is reached, the iteration is forcibly terminated regardless of whether the stopping condition is met, and the de-identified image sequence from the third iteration is used as the final output. In each iteration, the specific calculation of the pass rate is performed immediately after step S4, and the result is stored for comparison in the next iteration.

[0137] Step S6 is followed by:

[0138] Step S7: Use a lightweight convolutional neural network to extract image feature maps from the final de-identified image sequence, and use a lightweight pre-trained language model to extract text semantic vectors from the text records;

[0139] Step S8: Generate a random orthogonal matrix. Multiply the image feature map with the random orthogonal matrix to obtain the perturbed image features. The random orthogonal matrix satisfies the condition that its transpose multiplied by itself is an identity matrix.

[0140] Step S9: A gated attention fusion layer is used to fuse the perturbed image features with the text semantic vector to obtain a fused feature vector;

[0141] Step S10: Transmit the final de-identified image sequence using stream encryption, transmit the fused feature vector using homomorphic encryption, and transmit the text record using attribute encryption.

[0142] It should be further explained that in step S7, for each image in the final de-identified image sequence output from step S6, a lightweight convolutional neural network is input for feature extraction. This network adopts the MobileNetV3-Large structure and undergoes channel pruning with a pruning ratio of 0.3. The final number of convolutional channels retained are 16, 24, 40, 80, 112, and 160, respectively, and the network parameter count is 3.2 million. The input image size is 224×224 pixels, and the output image feature map... Its spatial dimension is 7×7 and the number of channels is 160. The feature map is compressed into a one-dimensional vector through global average pooling, resulting in an image feature vector of length 160.

[0143] For the corresponding text record, the lightweight pre-trained language model Distil BERT is used to extract the text semantic vector. This model undergoes four-bit quantization, compressing the original 66 million parameters to approximately 8 million parameters, with memory usage controlled within 80 megabytes. The maximum length of the input text record is limited to 128 words, and the 768-dimensional vector corresponding to the [CLS] marker in the model output is taken as the text semantic vector. .

[0144] In step S8, a random orthogonal matrix P with a size of 160×160 is generated. The orthogonal matrix is ​​generated as follows: first, a 160×160 matrix is ​​randomly generated, where each element follows a Gaussian distribution with mean 0 and variance 1; then, QR decomposition is performed on this matrix to obtain an orthogonal matrix Q and an upper triangular matrix R. Q is taken as the random orthogonal matrix P, satisfying the condition that the transpose of P multiplied by P equals the identity matrix. The image feature vector is then... Multiplying by the orthogonal matrix P yields the perturbed image features. The result is still a 160-dimensional vector. This orthogonal transformation is reversibly recoverable while holding the key P; the recovery process involves calculating... The transpose of .

[0145] In step S9, a gated attention fusion layer is used to process the perturbed image features. With text semantic vectors To merge them, first transform each of the two vectors to the same dimension d=256 using a linear transformation:

[0146] Mapped image features Mapped text features ,in, It is a 256×160 matrix. It is a 256×768 matrix. and The bias vector is 256-dimensional. Then, the attention gating coefficients are calculated:

[0147]

[0148] in, : Gated attention coefficient vector, each component is between 0 and 1; : Sigmoid function; The weight matrix of the gate layer is a 256×512 matrix; The bias vector of the gated layer is a 256-dimensional bias vector. The concatenated vector of image features and text features is a 512-dimensional vector. The mapped image and text features are both 256-dimensional vectors; [· ·] indicates concatenation in dimensions, resulting in a 512-dimensional vector;

[0149] Finally, calculate the fused feature vector:

[0150]

[0151] in, The final fused feature vector is 256-dimensional. : Gated attention coefficient vector; Element-wise multiplication; , : Mapped image features and text features.

[0152] In step S10, the final de-identified image sequence is transmitted using stream encryption, specifically the ChaCha20 algorithm with a 256-bit key length and a randomly generated 96-bit initialization vector. This encryption yields the ciphertext image data. The feature vectors are then fused. The transmission uses homomorphic encryption, specifically the BFV fully homomorphic encryption scheme. First, a public key and a private key are generated. Then, the public key is used to transmit the data. Each component is encoded as an integer polynomial and then encrypted to obtain a ciphertext feature vector. The original text record is transmitted using attribute encryption, specifically an attribute encryption scheme based on a ciphertext policy. The access policy is set to "doctor or researcher," and the text record is encrypted using the public key corresponding to this policy to obtain ciphertext text. The above three types of ciphertext data are packaged and uploaded to a cloud server or storage node. The private keys for homomorphic encryption and attribute encryption are stored locally by the security module and are not uploaded with the data.

[0153] This proposed method for privacy de-identification of medical multimodal image data, through the construction of a closed-loop architecture combining multimodal joint sensitive information detection and bimodal adversarial verification, can simultaneously eliminate traceable features such as explicit text identifiers and implicit device fingerprints in images. The three-level heterogeneous detection combined with cross-modal semantic alignment ensures that the identification of sensitive information does not rely on single-modal cues, effectively avoiding the missed detection problems associated with relying solely on images or text.

[0154] In bimodal adversarial verification, the responsive attack network and the feature residue detection network verify the thoroughness of de-identification from the perspectives of active attack and passive analysis, respectively. The joint judgment of the two ensures that the passage of either one is insufficient to be considered the final conclusion of privacy and security, thus forming a dual guarantee.

[0155] The meta-learning-driven recurrent self-feedback mechanism feeds back the information of samples that fail verification to the detection network and de-identification parameters, enabling the system to specifically strengthen the sensitivity and processing intensity of weak areas in each iteration. After a limited number of iterations, a stable and reliable de-identification effect can be achieved.

[0156] Furthermore, this method ensures strong privacy protection while also considering the feasibility of practical deployment and the integrity of diagnostic functions. The lightweight feature extraction network and reversible orthogonal perturbation mechanism enable the complete preservation of key anatomical features in the images after de-identification, and the perturbed features can be reversibly recovered when the key is held, thus balancing privacy protection and diagnostic traceability.

[0157] The gated attention fusion layer achieves adaptive fusion of image features and text semantics, obtaining effective joint feature representations without the need for complex neural networks. A hierarchical transmission strategy employing stream encryption, homomorphic encryption, and attribute encryption is used for different data types, satisfying both the requirements of cloud-based encrypted inference and enabling role-based access control.

[0158] The overall solution can complete preprocessing and iterative verification on edge devices, reducing reliance on cloud computing. At the same time, the closed-loop feedback mechanism ensures the consistency of multiple processing results, providing a repeatable and verifiable technical path for the privacy protection of medical multimodal data.

[0159] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0160] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for privacy de-identification processing of medical multimodal image data, characterized in that, Includes the following steps: Step S1: Obtain multimodal raw data, which includes medical image sequences and corresponding text records; Step S2: Perform the first round of multimodal joint sensitive information detection, detect explicit and implicit marker regions in each medical image, and output the set of sensitive regions and the comprehensive sensitivity of each sensitive region; Step S3: Perform adaptive de-identification processing on the sensitive regions based on the comprehensive sensitivity of each sensitive region to obtain the first round of de-identified image; Step S4: Perform bimodal adversarial verification on the first round of de-identified images. The bimodal adversarial verification includes: using a regenerative attack network to attempt to recover sensitive information from the first round of de-identified images and calculating the recovery success rate, and using a feature residue detection network to calculate the probability of residual sensitive features. When the recovery success rate is not greater than the first threshold and the probability of residual sensitive features is not greater than the second threshold, the verification is deemed to pass; otherwise, the verification is deemed to fail and the coordinates of the failed area are recorded. Step S5: Using the de-identified images that fail verification and their failure region coordinates as feedback signals, update the detection network parameters in step S2 and the de-identification parameters in step S3 through meta-learning. Step S6: Repeat steps S2 to S5 at least once until the pass rate of two consecutive rounds of verification is not less than the third threshold and the fluctuation range of the pass rate is not greater than the fourth threshold. Stop the iteration and output the final de-identified image sequence.

2. The method for privacy de-identification processing of medical multimodal image data according to claim 1, characterized in that: The multimodal joint sensitive information detection in step S2 includes: Level 1 detection: A lightweight texture-text separation network is used to quickly scan medical images and output a binary mask of candidate text regions; Second-level detection: After cropping the region indicated by the binary mask, the data is input into an attention-enhanced character recognition network, which outputs candidate sensitive entities, their confidence scores, and bounding boxes. Level 3 detection: Perform cross-modal semantic alignment between the candidate sensitive entities and the entities extracted by named entity recognition in the text records, mark only the successfully aligned entities as sensitive information, and calculate the explicit labeling sensitivity based on the alignment similarity; Simultaneously, medical images are input into an implicit identifier feature extraction network, which outputs an implicit sensitivity score. The explicit sensitivity score and the implicit sensitivity score are nonlinearly fused to obtain a comprehensive sensitivity score.

3. The method for privacy de-identification processing of medical multimodal image data according to claim 2, characterized in that: The cross-modal semantic alignment uses a BERT-based semantic similarity matrix to calculate the similarity between each candidate sensitive entity and each named entity in the text record. When the maximum similarity is greater than the fifth threshold, the alignment is considered successful, and the candidate sensitive entity is marked as sensitive information.

4. The method for privacy de-identification processing of medical multimodal image data according to claim 3, characterized in that: The adaptive de-identification process in step S3 includes: When the overall sensitivity of the sensitive area is greater than the sixth threshold, the pixel value is transformed by a reversible polynomial perturbation. The reversible polynomial perturbation is to perform a third-order polynomial operation on the pixel value in the encrypted domain, and the transformation coefficient is determined by the overall sensitivity. When the overall sensitivity is greater than the seventh threshold but not greater than the sixth threshold, adaptive Gaussian blur is used, and the size of the blur kernel is positively correlated with the overall sensitivity. When the overall sensitivity is no greater than the seventh threshold, only pixel-level Gaussian noise is injected.

5. The method for privacy de-identification processing of medical multimodal image data according to claim 4, characterized in that: The restorative attack network is a generator in a generative adversarial network. Its input is a de-identified image, and its output is a restored image. The restoration success rate is calculated as the ratio of the number of regions in the restored image and the original image whose structural similarity index at the coordinates of the failed region is greater than the eighth threshold to the total number of sensitive regions.

6. The method for privacy de-identification processing of medical multimodal image data according to claim 5, characterized in that: The feature residual detection network is a binary classification convolutional neural network. Its input is the difference image between the de-identified image and the original image, and its output is the probability value of residual sensitive features.

7. The method for privacy de-identification processing of medical multimodal image data according to claim 6, characterized in that: The meta-learning method in step S5 adopts a model-independent meta-learning framework. The total loss is obtained by weighted summation of the recovery success rate and residual sensitive feature probability of the failed samples in the previous iteration. Gradients are calculated for the detection network parameters and the de-identification parameters, and further or multi-step gradient updates are performed. The detection network parameters include the weights of the texture-text separation network, the weights of the attention-enhanced character recognition network, and the weights of the implicit identifier feature extraction network. The de-identification parameters include the polynomial coefficients in the invertible polynomial perturbation, the kernel size of the adaptive Gaussian blur, and the variance of the pixel-level Gaussian noise.

8. The method for privacy de-identification processing of medical multimodal image data according to claim 7, characterized in that: The feedback signal in step S5 also includes a structural difference map of the recovered image and the original image in the failure region, which is output by the recovery attack network to verify the failed sample. The structural difference map is used to guide the detection network to improve the detection sensitivity of the corresponding position of the failure region and its neighboring region in the next iteration.

9. A method for privacy de-identification processing of medical multimodal image data according to claim 8, characterized in that: The specific stop iteration condition in step S6 is as follows: the pass rate of two consecutive rounds of verification is not less than the ninth threshold, and the absolute value of the difference between the pass rate of the next round and the pass rate of the previous round is not greater than the tenth threshold; wherein the pass rate is equal to the ratio of the number of medical images that have passed verification to the total number of medical images processed in the current round.

10. A method for privacy de-identification processing of medical multimodal image data according to claim 9, characterized in that: Following step S6, the following is also included: Step S7: Use a lightweight convolutional neural network to extract image feature maps from the final de-identified image sequence, and use a lightweight pre-trained language model to extract text semantic vectors from the text records; Step S8: Generate a random orthogonal matrix, multiply the image feature map by the random orthogonal matrix to obtain the perturbed image features, wherein the random orthogonal matrix satisfies that its transpose multiplied by itself is an identity matrix; Step S9: A gated attention fusion layer is used to fuse the perturbed image features with the text semantic vector to obtain a fused feature vector; Step S10: The final de-identified image sequence is transmitted using stream encryption, the fused feature vector is transmitted using homomorphic encryption, and the text record is transmitted using attribute encryption.

Images