Multi-modal medical data encryption method and device based on large model, equipment and storage medium

Abstract

The invention discloses a multi-modal medical data encryption method and device based on a large model, equipment and a storage medium, relates to the field of artificial intelligence, is applied to a cloud service layer carrying a target labeling large model, and comprises the steps of obtaining original multi-modal medical data and corresponding target identity data sent by edge end equipment corresponding to the cloud service layer; performing feature extraction on the target identity data to obtain a corresponding biological feature vector, generating a target dynamic key based on the biological feature vector by using a target dynamic key generation algorithm, and encrypting the original multi-modal medical data by using the target dynamic key to obtain corresponding encrypted medical data; and obtaining a target labeling result corresponding to the encrypted medical data by using the target labeling large model, and binding the biological feature vector with the target labeling result by using the target dynamic key so as to encrypt and store the target labeling result in a target database. According to the method and the device, the data annotation efficiency is improved while the privacy disclosure risk is reduced.

Description

Multimodal medical data encryption methods, devices, equipment, and storage media based on large models Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, apparatus, device, and storage medium for encrypting multimodal medical data based on large models. Background Technology

[0002] With the widespread application of multimodal data (such as medical images and medical record text) in AI training, the privacy and efficiency of data annotation have become core pain points: multimodal data contains sensitive information such as faces, ID numbers, and medical records. In the traditional annotation process, data transmission and storage are easily stolen, and desensitization processing (such as blurring) often leads to data distortion, resulting in a prominent risk of privacy leakage. At the same time, cross-modal feature alignment (such as matching text descriptions with image content) requires the intervention of professionals, the dependence on manual annotation is high, and the annotation of a single data point can take up to minutes, which is extremely inefficient.

[0003] In conclusion, how to improve the efficiency of data annotation while reducing the risk of privacy leaks is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, apparatus, device and storage medium for encrypting multimodal medical data based on a large model, which can improve the efficiency of data annotation while reducing the risk of privacy leakage. The specific solution is as follows: Firstly, this application discloses a multimodal medical data encryption method based on a large model, applied to a cloud service layer equipped with a target annotation large model, including: acquiring raw multimodal medical data and corresponding target identity data sent by an edge device corresponding to the cloud service layer; the raw multimodal medical data includes medical image data and medical record text data input through the edge device; the target identity data includes collected fingerprint data and iris data of medical personnel; performing feature extraction on the target identity data to obtain corresponding biometric vectors, and using a target dynamic key generation algorithm to generate a target dynamic key based on the biometric vectors, so as to encrypt the raw multimodal medical data using the target dynamic key to obtain corresponding encrypted medical data; using the target annotation large model to obtain the target annotation result corresponding to the encrypted medical data, and using the target dynamic key to bind the biometric vectors with the target annotation result, so as to encrypt and save the target annotation result to a target database; the target annotation large model is a large model built based on the Transformer architecture, and the target database is a distributed database.

[0005] Optionally, the step of generating a target dynamic key based on the biometric vector using a target dynamic key generation algorithm includes: determining the biometric vector as an initial chaotic seed, and generating a target dynamic key based on the initial chaotic seed using a chaotic encryption algorithm.

[0006] Optionally, the multimodal medical data encryption method based on a large model further includes: obtaining new target identity data based on a preset key update cycle, and updating the target dynamic key based on the new target identity data using a target hash function to obtain a new target dynamic key; updating the regularization term of the target labeled large model based on the new target dynamic key to obtain the updated target labeled large model.

[0007] Optionally, encrypting the original multimodal medical data using the target dynamic key to obtain corresponding encrypted medical data includes: performing timestamp alignment and semantic mapping processing on the original multimodal medical data using the target labeled large model to obtain aligned medical data; processing the aligned medical data based on preset data augmentation operations to obtain corresponding enhanced medical data; and processing the enhanced medical data based on a preset differential privacy perturbation mechanism to obtain corresponding target medical data. The preset data augmentation operations include cropping and brightness adjustment of image data in the aligned medical data, synonym replacement of text data in the aligned medical data, and temporal stretching of audio data in the aligned medical data. The target dynamic key is used to encrypt image-sensitive data in the target medical data using a first encryption algorithm, and the target dynamic key is used to encrypt text-sensitive data in the target medical data using a second encryption algorithm to obtain the encrypted medical data corresponding to the target medical data.

[0008] Optionally, obtaining the target annotation result corresponding to the encrypted medical data using the target annotation large model includes: obtaining the first annotation result corresponding to the text data in the encrypted medical data using the first annotation branch of the target annotation large model; obtaining the second annotation result corresponding to the image data in the encrypted medical data using the second annotation branch of the target annotation large model; obtaining the third annotation result corresponding to the audio data in the encrypted medical data using the third annotation branch of the target annotation large model; and fusing the first annotation result, the second annotation result, and the third annotation result based on a preset cross-modal matching loss using the cross-modal fusion layer of the target annotation large model to obtain the target annotation result corresponding to the encrypted medical data.

[0009] Optionally, after binding the biometric vector with the target annotation result using the target dynamic key to encrypt and save the target annotation result to the target database, the method further includes: obtaining an annotation viewing request through a target visualization interface, performing biometric verification based on the identity data in the annotation viewing request, and determining the operation permissions corresponding to the request initiator of the annotation viewing request based on the verification result corresponding to the biometric verification; the operation permissions include annotation result viewing permissions and annotation result correction permissions.

[0010] Optionally, the multimodal medical data encryption method based on a large model further includes: using the target database to save the target parameters corresponding to the target annotation results stored in the target database to the target blockchain; the target parameters include the annotation ID, the hash digest corresponding to the biometric vector, and the timestamp; the target blockchain is a blockchain based on a consortium blockchain architecture, and the target nodes of the target blockchain include data holder nodes, annotation nodes, and verification nodes, and the consensus mechanism of the target blockchain is a consensus mechanism built based on a practical Byzantine fault-tolerant algorithm; and using zero-knowledge proof to verify the integrity of the target annotation results based on the target parameters stored in the target blockchain.

[0011] Secondly, this application discloses a multimodal medical data encryption device based on a large model, applied to a cloud service layer equipped with a target annotation large model, comprising: a data acquisition module, used to acquire raw multimodal medical data and corresponding target identity data sent by an edge device corresponding to the cloud service layer; the raw multimodal medical data includes medical image data and medical record text data input through the edge device; the target identity data includes collected fingerprint data and iris data of medical personnel; a data encryption module, used to extract features from the target identity data to obtain corresponding biometric vectors, and use a target dynamic key generation algorithm to generate a target dynamic key based on the biometric vectors, so as to encrypt the raw multimodal medical data using the target dynamic key to obtain corresponding encrypted medical data; and an annotation result storage module, used to obtain the target annotation result corresponding to the encrypted medical data using the target annotation large model, and use the target dynamic key to bind the biometric vectors with the target annotation result, so as to encrypt and save the target annotation result to a target database; the target annotation large model is a large model built based on the Transformer architecture, and the target database is a distributed database.

[0012] Thirdly, this application discloses an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the aforementioned multimodal medical data encryption method based on a large model.

[0013] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned multimodal medical data encryption method based on a large model.

[0014] In this application, when encrypting multimodal medical data, a cloud service layer equipped with a target annotation large model obtains the original multimodal medical data and corresponding target identity data sent by the edge device corresponding to the cloud service layer; the original multimodal medical data includes medical image data and medical record text data input through the edge device; the target identity data includes collected fingerprint data and iris data of medical personnel; feature extraction is performed on the target identity data to obtain corresponding biometric vectors, and a target dynamic key generation algorithm is used to generate a target dynamic key based on the biometric vectors, so as to encrypt the original multimodal medical data using the target dynamic key to obtain the corresponding encrypted medical data; the target annotation large model is used to obtain the target annotation result corresponding to the encrypted medical data, and the target dynamic key is used to bind the biometric vectors with the target annotation result to encrypt and save the target annotation result to the target database; the target annotation large model is a large model built based on the Transformer architecture, and the target database is a distributed database. As can be seen, this application achieves the binding of medical data with the identity data of medical personnel by converting the biometric vector corresponding to the target identity data into a target dynamic key, and then using the target dynamic key to encrypt the original multimodal medical data to obtain the corresponding encrypted medical data. This solves the problem of easy leakage of static keys and effectively protects privacy data. Simultaneously, it utilizes a target annotation large model to automatically annotate the encrypted medical data, reducing manual intervention and significantly improving data annotation efficiency. After annotation is completed, the annotation results are encrypted using the target dynamic key and saved to the target database, realizing dynamic verification and traceability of annotation permissions to prevent unauthorized use. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 is a flowchart of a multimodal medical data encryption method based on a large model disclosed in this application; Figure 2 is a schematic diagram of a specific multimodal medical data processing system architecture disclosed in this application; Figure 3 is a flowchart of a specific multimodal medical data encryption method based on a large model disclosed in this application; Figure 4 is a schematic diagram of a multimodal medical data encryption device based on a large model disclosed in this application; Figure 5 is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] With the widespread application of multimodal data (such as medical images and medical record text) in AI training, the privacy, security, and efficiency of data annotation have become core pain points. Multimodal data contains sensitive information such as faces, ID numbers, and medical records. Traditional annotation processes are prone to data theft during transmission and storage, and desensitization (such as blurring) often leads to data distortion, resulting in significant privacy risks. Simultaneously, cross-modal feature alignment (such as matching text descriptions with image content) requires professional intervention, resulting in high reliance on manual annotation, and the annotation time for a single data entry can reach minutes, leading to extremely low efficiency. To address these technical issues, this application discloses a multimodal medical data encryption method based on a large model, which can improve data annotation efficiency while reducing the risk of privacy leaks.

[0019] Referring to Figure 1, this embodiment of the invention discloses a multimodal medical data encryption method based on a large model, applied to a cloud service layer equipped with a target-labeled large model, including: step S11, obtaining the original multimodal medical data and corresponding target identity data sent by the edge device corresponding to the cloud service layer; the original multimodal medical data includes medical image data and medical record text data input through the edge device; the target identity data includes the collected fingerprint data and iris data of medical personnel.

[0020] In this embodiment, to achieve efficient processing and annotation of the data to be labeled, a multimodal medical data encryption system can be constructed based on a large target model. The system architecture diagram is shown in Figure 2. The edge devices in the edge terminal layer are responsible for inputting raw multimodal medical data, collecting biometric data, and performing local data preprocessing. The raw multimodal medical data includes medical image data and medical record text data input by medical personnel. The collected biometric data is the target identity data corresponding to the raw multimodal medical data, including the fingerprint and iris data of medical personnel.

[0021] Step S12: Extract features from the target identity data to obtain the corresponding biometric vector, and use a target dynamic key generation algorithm to generate a target dynamic key based on the biometric vector, so as to use the target dynamic key to encrypt the original multimodal medical data to obtain the corresponding encrypted medical data.

[0022] In this embodiment, as shown in Figure 3, feature extraction is performed on the target identity data to obtain the corresponding biometric vector. Specifically, this may include: collecting user fingerprints (optical sensor, 500 dpi resolution), iris scans (near-infrared imaging, 30 fps sampling rate), and removing noise using Gaussian filtering; extracting feature points: extracting minutiae features (endpoints, bifurcation points) from fingerprints, and extracting texture features from irises (using a Gabor filter), generating a 128-dimensional feature vector as the corresponding biometric vector. The extracted biometric vector will be converted into a dynamic key for data encryption and model constraints, solving the problem of static key leakage.

[0023] In one specific implementation, generating a target dynamic key based on a biometric vector using a target dynamic key generation algorithm includes: determining the biometric vector as an initial chaotic seed, and generating the target dynamic key based on the initial chaotic seed using a chaotic encryption algorithm. For example, a chaotic encryption algorithm (Logistic mapping, parameter...) can be used. =3.9), using the biometric vector as the initial chaotic seed, a 256-bit dynamic key is generated as the target dynamic key. Furthermore, to ensure the timeliness of the key, this embodiment can also acquire new target identity data based on a preset key update cycle, and use a target hash function to update the target dynamic key based on the new target identity data to obtain a new target dynamic key. For example, biometric increments (such as local fingerprint deformation) are collected every 30 minutes, and the key is updated using a hash function (SHA-256), thereby constructing a real-time key update mechanism, further reducing the risk of key leakage while ensuring key timeliness. Simultaneously, in this embodiment, the regularization term of the target labeled large model can also be updated based on the new target dynamic key to obtain the updated target labeled large model. By adding the correlation constraint between the key hash and model parameters to the loss function, it can be ensured that the model only works on data encrypted with a legitimate key. The target labeled large model is a large model capable of self-supervised learning built based on the Transformer architecture.

[0024] In this embodiment, the original multimodal medical data is encrypted using a target dynamic key to obtain corresponding encrypted medical data. This includes: using a target-annotated large model to perform timestamp alignment and semantic mapping on the original multimodal medical data to obtain aligned medical data; processing the aligned medical data based on preset data augmentation operations to obtain corresponding augmented medical data; and processing the augmented medical data based on a preset differential privacy perturbation mechanism to obtain corresponding target medical data. The preset data augmentation operations include cropping and brightness adjustment of image data in the aligned medical data, synonym replacement (e.g., based on WordNet) of text data in the aligned medical data, and temporal stretching of audio data in the aligned medical data. The training samples can also be further expanded. The target dynamic key is used to encrypt image-sensitive data in the target medical data based on a first encryption algorithm, and the target dynamic key is used to encrypt text-sensitive data in the target medical data based on a second encryption algorithm to obtain the encrypted medical data corresponding to the target medical data.

[0025] In one specific implementation, timestamp alignment processing primarily targets video and audio data in the original multimodal medical data, while semantic mapping processing mainly targets text and image data. This can be achieved by using cosine similarity matching between text vectors extracted by BERT and image vectors extracted by ResNet. When using the target dynamic key to perform layered encryption on the target medical data, AES-256 encryption can be used for image-sensitive regions (such as faces), and homomorphic encryption (Paillier algorithm) can be used for text-sensitive fields (such as medical record numbers), while retaining annotable features.

[0026] In one specific implementation, the enhanced medical data is processed based on a preset differential privacy perturbation mechanism to obtain the corresponding target medical data. Specifically, this may include: injecting differential privacy noise (Laplace noise) into the text data. =1.0), sensitive words (such as "AIDS") are replaced with semantically similar vague words (such as "infectious disease"); for image data, k-anonymization is performed on sensitive regions (such as faces) (preserving contours and blurring features), while Gaussian noise is only added to non-sensitive regions (such as background). =0.01); differential privacy waveform adjustment (adding random noise with a signal-to-noise ratio of 15dB) is applied to the audio data to preserve speech semantic features. Based on this preset differential privacy perturbation mechanism, the protection of privacy data can be further enhanced.

[0027] Step S13: Use the target annotation large model to obtain the target annotation result corresponding to the encrypted medical data, and use the target dynamic key to bind the biometric vector with the target annotation result to encrypt and save the target annotation result to the target database; the target annotation large model is a large model built based on the Transformer architecture, and the target database is a distributed database.

[0028] In this embodiment, the target annotation large model is a self-supervised model. The base model adopts the Transformer architecture, the text branch uses BERT, the image branch uses ViT, and the audio branch uses Wav2Vec 2.0. Feature fusion is achieved through a shared attention layer. Simultaneously, this embodiment designs "cross-modal matching loss" (minimizing the loss when matching text descriptions with image content) and "intra-modal consistency loss" (minimizing the feature distance between different enhanced versions of the same image) as comparative learning tasks for the target annotation large model.

[0029] In one specific implementation, obtaining target annotation results corresponding to encrypted medical data using a large target annotation model includes: obtaining first annotation results corresponding to text data in the encrypted medical data using a first annotation branch of the large target annotation model; obtaining second annotation results corresponding to image data in the encrypted medical data using a second annotation branch of the large target annotation model; obtaining third annotation results corresponding to audio data in the encrypted medical data using a third annotation branch of the large target annotation model; and fusing the first annotation results, second annotation results, and third annotation results using a cross-modal fusion layer of the large target annotation model based on a preset cross-modal matching loss to obtain target annotation results corresponding to the encrypted medical data.

[0030] In practical applications, the target annotation large model predicts unlabeled encrypted medical data and generates pseudo-labels with a confidence level of ≥0.8 as the corresponding target annotation results (such as the association annotation of "tumor region in image" with "malignant tumor text in medical record"). After the annotation is completed, the current model parameters of the target annotation large model can be updated based on all historical pseudo-label data through the momentum comparison (MoCo) mechanism. After three iterations, the accuracy of the pseudo-labels output by the target annotation large model can be improved to more than 85%.

[0031] In this embodiment, a target dynamic key is used to bind biometric vectors with target annotation results to encrypt and save the target annotation results to the target database. This includes two parts: biometric hash generation and encrypted storage of annotation results. Specifically, biometric hash generation can include: using the locality-sensitive hashing (LSH) algorithm to map a 128-dimensional biometric vector to a 64-bit hash value, preserving local feature similarity; binding the hash value to the annotation ID: each annotation result generates a unique UUID, which is then fused with the biometric hash using an XOR operation to form an "annotation-identity" binding code. Encrypted storage of annotation results can include: the annotation results (such as bounding box coordinates and text entity labels) are encrypted with a dynamic key and stored in a distributed database (such as MongoDB) indexed as the "annotation-identity" binding code, which can only be queried through real-time biometric hash verification (Hamming distance ≤ 3).

[0032] In this embodiment, role-based access control (RBAC) can also be implemented based on biometrics: administrators can configure "annotators" to only view text labels and "auditors" to view image annotation details; modifying annotations requires secondary biometric verification (such as simultaneous fingerprint and iris verification), while viewing annotations only requires a single verification, thereby achieving granular control of permissions. In a specific implementation, after binding the biometric vector with the target annotation result using a target dynamic key to encrypt and save the target annotation result to the target database, the method further includes: obtaining an annotation viewing request through a target visualization interface, performing biometric verification based on the identity data in the annotation viewing request, and determining the operation permissions corresponding to the request initiator of the annotation viewing request based on the verification result corresponding to the biometric verification; the operation permissions include annotation result viewing permissions and annotation result correction permissions.

[0033] It should be noted that this embodiment also constructs a pseudo-label correction triggering mechanism. After a legitimate user passes biometric verification, the system calls the feature difference between the local original data (unperturbed) and the perturbed data (such as calculating the structural similarity index SSIM of the image). When the difference value is ≥0.2, the correction process is triggered, and a comparison sample pair is constructed using the original data and the perturbed data to train the model to learn the "perturbed bias". Based on the biometric verification-triggered correction mechanism, this embodiment resolves the contradiction between privacy and data accuracy. In a specific implementation, the correction loss function of the target labeling large model is: ;in =0.7, =0.3, For pseudo-label loss, The loss is the original-perturbation difference; in addition, the target annotation large model can be fine-tuned every hour using newly collected corrected samples (e.g., using the SGD optimizer with a learning rate of 1e-5), realizing online updates of the target annotation large model and ensuring stable annotation accuracy.

[0034] In this embodiment, the target parameters corresponding to the target annotation results stored in the target database can also be saved to the target blockchain using the target database. The target parameters include the annotation ID, the hash digest corresponding to the biometric vector, and the timestamp. The target blockchain is a blockchain based on a consortium blockchain architecture. The target nodes of the target blockchain include data holder nodes, annotation nodes, and reviewer nodes. The consensus mechanism of the target blockchain is a consensus mechanism built based on a practical Byzantine fault-tolerant algorithm. Zero-knowledge proofs are used to verify the integrity of the target annotation results based on the target parameters stored in the target blockchain. In one specific implementation, the target blockchain uses data holders (such as hospitals), annotation parties, and reviewers as nodes, adopts a consortium blockchain (Hyperledger Fabric) architecture, and uses practical Byzantine fault tolerance (PBFT) to ensure that consensus can still be reached even when 33% of the nodes are abnormal, with a block generation time of ≤5 seconds. When using zero-knowledge proofs for evidence storage, only the annotation ID, biometric hash digest, and timestamp are stored, without including the original data or annotation content. The annotation record is uploaded to the blockchain by storing key target parameters of the annotation result. The auditor then verifies the annotation integrity (e.g., "the annotation rate of this batch of data is ≥95%)" using zero-knowledge proofs (ZKP), achieving audit verification without exposing the specific content. Furthermore, in this embodiment, smart contracts and permission decay can be implemented based on the target blockchain. Specifically, a permission management contract is deployed by defining the annotator's biometric hash, annotation range (e.g., "chest CT image"), and permission validity (e.g., 24 hours). A time decay algorithm is used to update permission validity, causing it to decay linearly over time (e.g., permission value = 1 at t=0, permission value = 0 at t=24 hours), requiring re-verification of biometrics to refresh permissions.

[0035] In one specific implementation, the multimodal medical data encryption system constructed in this embodiment can realize visualized interaction and real-time feedback of the entire annotation process based on an interactive interface, supporting manual intervention and model iteration. Specifically, the visualization interface includes a multimodal data display area and a permission status area. The left side of the multimodal data display area displays the original data (requiring biometric verification), and the right side displays perturbed data and automatic annotation results (e.g., text entities are highlighted in red, and image bounding boxes are marked in blue); the permission status area displays the remaining time of the current permission, the biometric verification status, and provides warnings when permissions are insufficient. Annotators can correct pseudo-labels through the interface (e.g., adjusting image bounding box coordinates), and corrected records are automatically associated with biometric hashes and uploaded to the blockchain; feedback data can be used for model optimization, adding manually corrected samples (accounting for 10%) to the training set, and retraining the self-supervised model every 24 hours. The visualization interface can also monitor indicators in real time, including annotation accuracy (the matching rate between manually corrected and automatically labeled data) and privacy leakage risk (through differential privacy budgeting). Value monitoring), system response time; when the annotation accuracy is <90%, automatically increase the proportion of manual feedback samples; when When the value is less than 0.5, the intensity of data disturbance is reduced, thereby enabling dynamic optimization of the annotation mechanism for the target annotation large model.

[0036] It should be noted that in practical applications, this embodiment combines multiple encryption methods such as dynamic key encryption, differential perturbation, and blockchain notarization to reduce the risk of privacy data leakage by more than 90%, meeting compliance requirements such as GDPR and HIPAA, and significantly enhancing privacy protection capabilities. The self-supervised learning mechanism of the target annotation large model reduces manual annotation by 80%, shortening the annotation time for a single multimodal data point from 5 minutes to 30 seconds, significantly improving annotation efficiency. Furthermore, the pseudo-label correction closed-loop mechanism maintains the annotation accuracy of desensitized data at 92% (compared to only 70% for traditional methods), achieving synergistic optimization of annotation accuracy and data privacy protection. The biometric time decay mechanism reduces the risk of unauthorized access by 95%, supporting highly regulated scenarios such as healthcare and enabling refined access control. Blockchain notarization ensures that annotation records are tamper-proof and guarantees full traceability of the annotation record process.

[0037] As can be seen, this application achieves the binding of medical data with the identity data of medical personnel by converting the biometric vector corresponding to the target identity data into a target dynamic key, and then using the target dynamic key to encrypt the original multimodal medical data to obtain the corresponding encrypted medical data. This solves the problem of easy leakage of static keys and effectively protects privacy data. Simultaneously, it utilizes a target annotation large model to automatically annotate the encrypted medical data, reducing manual intervention and significantly improving data annotation efficiency. After annotation is completed, the annotation results are encrypted using the target dynamic key and saved to the target database, realizing dynamic verification and traceability of annotation permissions to prevent unauthorized use.

[0038] As described in the previous embodiment, this application discloses a multimodal medical data encryption method based on a large model, which can improve the efficiency of data annotation while reducing the risk of privacy leakage. Next, the implementation process of multimodal medical data encryption will be explained for a medical and healthcare scenario.

[0039] The multimodal medical data encryption system proposed in this embodiment is deployed in a hybrid cloud environment, including: (1) edge nodes: biometric acquisition terminals (fingerprinters, iris scanners), local data preprocessing modules (deployed in hospital / enterprise intranets to process raw data); (2) cloud nodes: self-supervised learning servers (GPU cluster, 8×NVIDIA A100), blockchain nodes (5 alliance nodes, distributed in different institutions), and distributed database (MongoDB sharded cluster).

[0040] When encrypting the input data, the core process of the multimodal medical data encryption system includes: (1) Data access stage: The hospital client uploads multimodal data (such as CT images + diagnostic text), the edge node automatically triggers biometric collection (doctor's fingerprint), and generates a dynamic key; after the data is encrypted, it is transmitted to the cloud, and only the annotable features such as "image outline + text semantic framework" are retained.

[0041] (2) Automatic labeling stage: The cloud-based self-supervised model performs cross-modal feature alignment on encrypted data (such as associating the "nodule" region in CT images with the text "lung nodule diameter 5mm"); generates initial pseudo-labels, verifies them through biometric hashes, and stores them, while triggering blockchain notarization (recording label ID, timestamp, doctor hash).

[0042] (3) Manual correction stage: Doctors unlock the annotation interface through iris verification, view the automatic annotation results, and correct the deviation labels (such as bounding box offset); the correction record is fed back to the model in real time, triggering local fine-tuning, and updating the blockchain evidence at the same time.

[0043] (4) Permission revocation phase: 24 hours after the annotation is completed, the system will automatically reduce the permission. Doctors need to re-verify their fingerprints before they can access the site again. If abnormal annotation is found (such as multiple corrections of the same label), the administrator can trace the specific operator through the blockchain and trigger the audit process.

[0044] As can be seen, this application achieves the binding of medical data with the identity data of medical personnel by converting the biometric vector corresponding to the target identity data into a target dynamic key, and then using the target dynamic key to encrypt the original multimodal medical data to obtain the corresponding encrypted medical data. This solves the problem of easy leakage of static keys and effectively protects privacy data. Simultaneously, it utilizes a target annotation large model to automatically annotate the encrypted medical data, reducing manual intervention and significantly improving data annotation efficiency. After annotation is completed, the annotation results are encrypted using the target dynamic key and saved to the target database, realizing dynamic verification and traceability of annotation permissions to prevent unauthorized use.

[0045] Referring to Figure 4, this application discloses a multimodal medical data encryption device based on a large model, applied to a cloud service layer equipped with a target annotation large model, comprising: a data acquisition module 11, used to acquire raw multimodal medical data and corresponding target identity data sent by an edge device corresponding to the cloud service layer; the raw multimodal medical data includes medical image data and medical record text data input through the edge device; the target identity data includes collected fingerprint data and iris data of medical personnel; a data encryption module 12, used to extract features from the target identity data to obtain corresponding biometric vectors, and use a target dynamic key generation algorithm to generate a target dynamic key based on the biometric vectors, so as to encrypt the raw multimodal medical data using the target dynamic key to obtain the corresponding encrypted medical data; and an annotation result storage module 13, used to obtain the target annotation result corresponding to the encrypted medical data using the target annotation large model, and use the target dynamic key to bind the biometric vectors with the target annotation result, so as to encrypt and save the target annotation result to a target database; the target annotation large model is a large model built based on the Transformer architecture, and the target database is a distributed database.

[0046] As can be seen, this application achieves the binding of medical data with the identity data of medical personnel by converting the biometric vector corresponding to the target identity data into a target dynamic key, and then using the target dynamic key to encrypt the original multimodal medical data to obtain the corresponding encrypted medical data. This solves the problem of easy leakage of static keys and effectively protects privacy data. Simultaneously, it utilizes a target annotation large model to automatically annotate the encrypted medical data, reducing manual intervention and significantly improving data annotation efficiency. After annotation is completed, the annotation results are encrypted using the target dynamic key and saved to the target database, realizing dynamic verification and traceability of annotation permissions to prevent unauthorized use.

[0047] In one specific embodiment, the data encryption module 12 may include: a key generation unit, used to determine the biometric vector as an initial chaotic seed, and use a chaotic encryption algorithm to generate a target dynamic key based on the initial chaotic seed.

[0048] In one specific embodiment, the apparatus may further include: a key update module, configured to acquire new target identity data based on a preset key update cycle, and update the target dynamic key based on the new target identity data using a target hash function to obtain a new target dynamic key; and a model update module, configured to update the regularization term of the target annotation large model based on the new target dynamic key to obtain the updated target annotation large model.

[0049] In one specific embodiment, the data encryption module 12 may include: a data processing unit, configured to perform timestamp alignment and semantic mapping processing on the original multimodal medical data using the target annotation large model to obtain aligned medical data, process the aligned medical data based on preset data augmentation operations to obtain corresponding augmented medical data, and process the augmented medical data based on a preset differential privacy perturbation mechanism to obtain corresponding target medical data; the preset data augmentation operations include cropping and brightness adjustment of image data in the aligned medical data, synonym replacement of text data in the aligned medical data, and temporal stretching of audio data in the aligned medical data; and a data encryption unit, configured to encrypt image-sensitive data in the target medical data using the target dynamic key based on a first encryption algorithm, and encrypt text-sensitive data in the target medical data using the target dynamic key based on a second encryption algorithm, to obtain encrypted medical data corresponding to the target medical data.

[0050] In one specific implementation, the annotation result storage module 13 may include: a first annotation unit, used to obtain a first annotation result corresponding to text data in the encrypted medical data using a first annotation branch of the target annotation model; a second annotation unit, used to obtain a second annotation result corresponding to image data in the encrypted medical data using a second annotation branch of the target annotation model; a third annotation unit, used to obtain a third annotation result corresponding to audio data in the encrypted medical data using a third annotation branch of the target annotation model; and an annotation fusion unit, used to fuse the first annotation result, the second annotation result, and the third annotation result based on a preset cross-modal matching loss using the cross-modal fusion layer of the target annotation model to obtain the target annotation result corresponding to the encrypted medical data.

[0051] In one specific embodiment, the device may further include: an access control module, configured to obtain a label viewing request through a target visualization interface, perform biometric verification based on the identity data in the label viewing request, and determine the operation permissions corresponding to the request initiator of the label viewing request based on the verification result corresponding to the biometric verification; the operation permissions include label result viewing permissions and label result correction permissions.

[0052] In one specific embodiment, the device may further include: a parameter on-chain module, used to save target parameters corresponding to the target annotation results stored in the target database to the target blockchain using the target database; the target parameters include annotation ID, hash digest corresponding to the biometric vector, and timestamp; the target blockchain is a blockchain based on a consortium blockchain architecture, the target nodes of the target blockchain include data holder nodes, annotation nodes, and verification nodes, and the consensus mechanism of the target blockchain is a consensus mechanism built based on a practical Byzantine fault-tolerant algorithm; and an annotation verification module, used to verify the integrity of the target annotation results based on the target parameters stored in the target blockchain using zero-knowledge proofs.

[0053] Furthermore, this application also discloses an electronic device. FIG5 is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the figure should not be considered as any limitation on the scope of use of this application.

[0054] Figure 5 is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the multimodal medical data encryption method based on a large model disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0055] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0056] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.

[0057] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the large-model-based multimodal medical data encryption method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0058] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed multimodal medical data encryption method based on a large model. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0059] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0060] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0061] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0062] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0063] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for encrypting multimodal medical data based on a large model, characterized in that, This method is applied to a cloud service layer equipped with a large-scale target annotation model, comprising: acquiring raw multimodal medical data and corresponding target identity data sent by edge devices corresponding to the cloud service layer; the raw multimodal medical data includes medical image data and medical record text data input through the edge devices; the target identity data includes collected fingerprint data and iris data of medical personnel; performing feature extraction on the target identity data to obtain corresponding biometric vectors, and generating a target dynamic key based on the biometric vectors using a target dynamic key generation algorithm, thereby encrypting the raw multimodal medical data using the target dynamic key to obtain corresponding encrypted medical data; using the large-scale target annotation model to obtain target annotation results corresponding to the encrypted medical data, and binding the biometric vectors with the target annotation results using the target dynamic key to encrypt and save the target annotation results to a target database; the large-scale target annotation model is a large model built based on the Transformer architecture, and the target database is a distributed database.

2. The multimodal medical data encryption method based on a large model according to claim 1, characterized in that, The method of generating a target dynamic key based on the biometric vector using the target dynamic key generation algorithm includes: determining the biometric vector as an initial chaotic seed, and generating a target dynamic key based on the initial chaotic seed using a chaotic encryption algorithm.

3. The multimodal medical data encryption method based on a large model according to claim 1, characterized in that, Also includes: The new target identity data is obtained based on a preset key update cycle, and the target dynamic key is updated based on the new target identity data using a target hash function to obtain a new target dynamic key. The regularization term of the target annotation model is updated based on the new target dynamic key to obtain the updated target annotation model.

4. The multimodal medical data encryption method based on a large model according to claim 1, characterized in that, The step of encrypting the original multimodal medical data using the target dynamic key to obtain corresponding encrypted medical data includes: performing timestamp alignment and semantic mapping processing on the original multimodal medical data using the target labeled large model to obtain aligned medical data; processing the aligned medical data based on preset data augmentation operations to obtain corresponding enhanced medical data; and processing the enhanced medical data based on a preset differential privacy perturbation mechanism to obtain corresponding target medical data. The preset data augmentation operations include cropping and brightness adjustment of image data in the aligned medical data, synonym replacement of text data in the aligned medical data, and temporal stretching of audio data in the aligned medical data; encrypting image-sensitive data in the target medical data using the target dynamic key based on a first encryption algorithm, and encrypting text-sensitive data in the target medical data using the target dynamic key based on a second encryption algorithm to obtain the encrypted medical data corresponding to the target medical data.

5. The multimodal medical data encryption method based on a large model according to claim 1, characterized in that, The step of obtaining the target annotation result corresponding to the encrypted medical data using the target annotation large model includes: obtaining the first annotation result corresponding to the text data in the encrypted medical data using the first annotation branch of the target annotation large model; obtaining the second annotation result corresponding to the image data in the encrypted medical data using the second annotation branch of the target annotation large model; obtaining the third annotation result corresponding to the audio data in the encrypted medical data using the third annotation branch of the target annotation large model; and fusing the first annotation result, the second annotation result, and the third annotation result based on a preset cross-modal matching loss using the cross-modal fusion layer of the target annotation large model to obtain the target annotation result corresponding to the encrypted medical data.

6. The multimodal medical data encryption method based on a large model according to claim 1, characterized in that, After binding the biometric vector with the target annotation result using the target dynamic key to encrypt and save the target annotation result to the target database, the method further includes: obtaining an annotation viewing request through a target visualization interface, performing biometric verification based on the identity data in the annotation viewing request, and determining the operation permissions corresponding to the request initiator of the annotation viewing request based on the verification result corresponding to the biometric verification; the operation permissions include annotation result viewing permissions and annotation result correction permissions.

7. The method for encrypting multimodal medical data based on a large model according to any one of claims 1 to 6, characterized in that, It also includes: using the target database to save the target parameters corresponding to the target annotation results stored in the target database to the target blockchain; the target parameters include the annotation ID, the hash digest corresponding to the biometric vector, and the timestamp; the target blockchain is a blockchain based on a consortium blockchain architecture, and the target nodes of the target blockchain include data holder nodes, annotation nodes, and verification nodes, and the consensus mechanism of the target blockchain is a consensus mechanism built based on a practical Byzantine fault-tolerant algorithm; and using zero-knowledge proof to verify the integrity of the target annotation results based on the target parameters stored in the target blockchain.

8. A multimodal medical data encryption device based on a large model, characterized in that, An application to a cloud service layer equipped with a large target annotation model includes: a data acquisition module for acquiring raw multimodal medical data and corresponding target identity data sent by edge devices corresponding to the cloud service layer; the raw multimodal medical data includes medical image data and medical record text data input through the edge devices; the target identity data includes collected fingerprint data and iris data of medical personnel; a data encryption module for extracting features from the target identity data to obtain corresponding biometric vectors, and generating a target dynamic key based on the biometric vectors using a target dynamic key generation algorithm, so as to encrypt the raw multimodal medical data using the target dynamic key to obtain the corresponding encrypted medical data; and an annotation result storage module for obtaining the target annotation result corresponding to the encrypted medical data using the large target annotation model, binding the biometric vectors with the target annotation result using the target dynamic key, so as to encrypt and save the target annotation result to a target database; the large target annotation model is a large model built based on the Transformer architecture, and the target database is a distributed database.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the multimodal medical data encryption method based on a large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs, wherein the computer programs, when executed by a processor, implement the multimodal medical data encryption method based on a large model as described in any one of claims 1 to 7.

Images