Diffusion model training method and system based on privacy calculation

Abstract

The invention provides a diffusion model training method and system based on privacy calculation in the technical field of artificial intelligence and data security. The method comprises the steps that S1, a client obtains input image data, text cues and setting information; s2, based on the setting information, extracting image features from the image data through encryption calculation, differential privacy calculation or federated learning technology, and encrypting the image features into privacy features; s3, transmitting the privacy features and the text prompt words to a server; s4, the server inputs the privacy features and the text prompt words into a diffusion model to obtain a bionic image; s5, labeling prompt word labels for the bionic images to construct a bionic image training set; and S6, optimizing the diffusion model through the bionic graph training set. The method has the advantages that the model generation capability is maintained on the premise of fully protecting data privacy, and a complete technical closed loop from data security processing to training set efficient construction is realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data security technology, and in particular to a diffusion model training method and system based on privacy computing. Background Technology

[0002] In recent years, diffusion models have demonstrated outstanding generation performance and broad application prospects in the field of image generation, and have been successfully applied to various task scenarios such as text-to-image and image-to-image.

[0003] Training diffusion models typically relies on large-scale, high-quality image-text pairing datasets. However, in practical applications, training data often contains content involving personal privacy or sensitive information, such as facial images and medical images. Directly using such data to train models poses a significant risk of data privacy breaches. To mitigate these risks, existing technologies often employ data anonymization and federated learning, but these methods still have the following shortcomings: 1. Traditional data anonymization methods have limited defense capabilities against re-identification attacks, making it difficult to completely prevent privacy leaks and potentially affecting the effectiveness of privacy protection and the reliability of related business decisions; 2. It is not easy to balance the strength of privacy protection with data availability. Over-protection can lead to a decline in the quality of training data, which in turn affects the generative performance of the diffusion model. 3. Existing solutions are mostly modular and decentralized, lacking integrated system support covering the entire process from privacy-preserving computation and synthetic image generation to high-quality training set construction.

[0004] Therefore, how to provide a diffusion model training method and system based on privacy computing, so as to maintain the model generation capability while fully protecting data privacy, and realize a complete technical closed loop from secure data processing to efficient construction of training sets, has become an urgent technical problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a diffusion model training method and system based on privacy computing, which can maintain the model generation capability while fully protecting data privacy, and realize a complete technical closed loop from data security processing to efficient construction of training sets.

[0006] In a first aspect, the present invention provides a diffusion model training method based on privacy computing, comprising the following steps: Step S1: The client obtains the input image data, text prompts, and settings information, and performs preprocessing on the image data, text prompts, and settings information, including formatting and cleaning. Step S2: Based on the settings information, the client extracts image features from each of the image data through encryption calculation, differential privacy calculation, or federated learning techniques, and encrypts each of the image features into corresponding privacy features. Step S3: The client transmits each of the privacy features and text prompts to the server, and the server stores the received privacy features and text prompts. Step S4: The server inputs the privacy features and text prompts into the pre-trained diffusion model to obtain a biomimetic image; Step S5: Extract the text theme from the text prompt words, extract the biomimetic image theme from the biomimetic image, and label each biomimetic image with prompt word tags based on the text theme and biomimetic image theme to construct a biomimetic image training set. Step S6: Continuously train and optimize the diffusion model using the biomimetic graph training set.

[0007] Furthermore, step S1 specifically includes: The client acquires input image data, text prompts, and settings information. The settings information carries a privacy protection method, which may be encrypted computation, differential privacy computation, or federated learning. When the privacy protection method is encrypted computation, the settings information also carries an encryption algorithm. When the privacy protection method is differential privacy computation, the settings information also carries the noise level of the differential privacy computation. When the privacy protection method is federated learning, the settings information also carries the configuration of federated learning participants. The image data is formatted by at least the following processes: size adjustment, color space conversion, data normalization, and format conversion. The formatted image data is then cleaned by at least the following processes: quality filtering, duplicate detection, outlier handling, and content filtering, thereby completing the preprocessing of the image data. The text prompts are formatted by at least text normalization, word segmentation, encoding conversion and length standardization. The formatted text prompts are then cleaned by at least spell correction, stop word removal, irrelevant content filtering and language recognition, thereby completing the preprocessing of the text prompts. The settings information is formatted by at least structural parsing, parameter standardization, and default value filling. The formatted settings information is then cleaned by at least validity verification, security checks, and consistency checks, thereby completing the preprocessing of the settings information.

[0008] Furthermore, step S2 specifically includes: Based on the privacy protection method carried in the settings information, the client extracts image features from each of the image data in real time through encrypted computation, differential privacy computation, or federated learning techniques, and encrypts each of the image features into corresponding privacy features through homomorphic encryption algorithms.

[0009] Furthermore, step S3 specifically includes: The client transmits each privacy feature and text prompt to the server through a secure channel. The server receives each privacy feature and text prompt in real time and stores each privacy feature and text prompt in the hardware security module.

[0010] Furthermore, step S4 specifically includes: The server inputs the privacy features and text prompts into a pre-trained diffusion model. The diffusion model performs inference using hardware acceleration technology to obtain a biomimetic image that is semantically consistent with the image data corresponding to the privacy features and does not contain user privacy information. Step S5 specifically involves: The text topics are extracted from the text prompts using a pre-trained KeyBERT model, and the biomimetic image topics are extracted from the biomimetic images using a pre-trained BLIP-2 model. Based on the text topics and biomimetic image topics, the prompts are automatically labeled for each biomimetic image using a pre-trained multimodal large model to construct a biomimetic image training set. Secondly, the present invention provides a diffusion model training system based on privacy computing, comprising the following modules: The data input preprocessing module is used by the client to obtain input image data, text prompts, and setting information, and to perform preprocessing on the image data, text prompts, and setting information, including formatting and cleaning. The privacy feature extraction module is used by the client to extract image features from each of the image data based on the setting information through encryption calculation, differential privacy calculation or federated learning technology, and encrypt each of the image features into corresponding privacy features; The privacy feature upload module is used by the client to transmit each privacy feature and text prompt to the server, and the server stores the received privacy features and text prompt. A biomimetic image generation module is used by the server to input the privacy features and text prompts into a pre-trained diffusion model to obtain a biomimetic image; The biomimetic image training set construction module is used to extract text topics from the text prompts, extract biomimetic image topics from the biomimetic images, and label each biomimetic image with prompts based on the text topics and biomimetic image topics in order to construct a biomimetic image training set. The diffusion model training module is used to continuously train and optimize the diffusion model using the biomimetic graph training set.

[0011] Furthermore, the data input preprocessing module is specifically used for: The client acquires input image data, text prompts, and settings information. The settings information carries a privacy protection method, which may be encrypted computation, differential privacy computation, or federated learning. When the privacy protection method is encrypted computation, the settings information also carries an encryption algorithm. When the privacy protection method is differential privacy computation, the settings information also carries the noise level of the differential privacy computation. When the privacy protection method is federated learning, the settings information also carries the configuration of federated learning participants. The image data is formatted by at least the following processes: size adjustment, color space conversion, data normalization, and format conversion. The formatted image data is then cleaned by at least the following processes: quality filtering, duplicate detection, outlier handling, and content filtering, thereby completing the preprocessing of the image data. The text prompts are formatted by at least text normalization, word segmentation, encoding conversion and length standardization. The formatted text prompts are then cleaned by at least spell correction, stop word removal, irrelevant content filtering and language recognition, thereby completing the preprocessing of the text prompts. The settings information is formatted by at least structural parsing, parameter standardization, and default value filling. The formatted settings information is then cleaned by at least validity verification, security checks, and consistency checks, thereby completing the preprocessing of the settings information.

[0012] Furthermore, the privacy feature extraction module is specifically used for: Based on the privacy protection method carried in the settings information, the client extracts image features from each of the image data in real time through encrypted computation, differential privacy computation, or federated learning techniques, and encrypts each of the image features into corresponding privacy features through homomorphic encryption algorithms.

[0013] Furthermore, the privacy feature uploading module is specifically used for: The client transmits each privacy feature and text prompt to the server through a secure channel. The server receives each privacy feature and text prompt in real time and stores each privacy feature and text prompt in the hardware security module.

[0014] Furthermore, the biomimetic image generation module is specifically used for: The server inputs the privacy features and text prompts into a pre-trained diffusion model. The diffusion model performs inference using hardware acceleration technology to obtain a biomimetic image that is semantically consistent with the image data corresponding to the privacy features and does not contain user privacy information. The biomimetic graph training set construction module is specifically used for: The text topics are extracted from the text prompts using a pre-trained KeyBERT model, and the biomimetic image topics are extracted from the biomimetic images using a pre-trained BLIP-2 model. Based on the text topics and biomimetic image topics, the prompts are automatically labeled for each biomimetic image using a pre-trained multimodal large model to construct a biomimetic image training set.

[0015] The advantages of this invention are: 1. The system acquires input image data, text prompts, and settings information from the client. Preprocessing of these data, including formatting and cleaning, is performed. Then, based on the settings information, image features are extracted from each image data using encryption computation, differential privacy computation, or federated learning techniques. Each image feature is encrypted into a corresponding privacy feature, and these privacy features, along with the text prompts, are transmitted to the server. The server stores the received privacy features and text prompts. Next, the server inputs these privacy features and text prompts into a pre-trained diffusion model to obtain biomimetic images. Text topics are extracted from the text prompts, and biomimetic image topics are extracted from the biomimetic images. Based on the text topics and biomimetic image topics, prompt labels are applied to each biomimetic image to construct a biomimetic image training set. Finally, the diffusion model is trained using the biomimetic image training set. The diffusion model undergoes continuous training and optimization. Specifically, on the client side, privacy-preserving computation techniques (such as cryptographic computation, differential privacy, federated learning, and homomorphic encryption algorithms) are used to transform raw image data into protected privacy features, ensuring strict protection of the original data privacy at the source. On the server side, a pre-trained diffusion model is used to generate semantically consistent but entirely new biomimetic images based on these privacy features and text prompts. This transforms the conflict between privacy protection and data usability into the problem of generating high-quality new data using secure data. Furthermore, an automated model extracts and labels the themes of the text and biomimetic images, efficiently constructing a high-quality biomimetic graph training set, which is ultimately used for continuous optimization of the diffusion model. This achieves the goal of maintaining model generation capabilities while fully protecting data privacy, and realizing a complete technical loop from secure data processing to efficient training set construction.

[0016] 2. By integrating encrypted computing, differential privacy computing, or federated learning techniques, privacy features are extracted and encrypted from image data during the training of the diffusion model. This ensures that the original user data is not exposed to the server in plaintext. This method effectively prevents the risk of privacy leakage, complies with increasingly stringent data protection regulations (such as GDPR), and maintains user trust. It is particularly suitable for data processing scenarios in sensitive fields such as healthcare and finance.

[0017] 3. By employing secure channels and hardware security modules in the data transmission and storage stages, combined with formatting and cleaning in the preprocessing steps, we ensured that image and text data were protected from theft or tampering during transmission and eliminated interference from low-quality or abnormal data. This end-to-end security design reduced the risk of data contamination, improved the reliability of training data, and laid a solid foundation for model optimization.

[0018] 4. By automating the preprocessing, feature extraction, and annotation processes, the need for manual intervention is reduced. For example, KeyBERT and BLIP-2 models are used for topic extraction and automatic annotation. This not only accelerates the construction of the biomimetic graph training set, but also improves the inference speed of the diffusion model through hardware acceleration technology, thus shortening the overall model training cycle and making it suitable for large-scale real-time applications.

[0019] 5. By utilizing multimodal ensemble, text prompts are combined with image features, and a high-quality training set is constructed through biomimetic images. This enables the diffusion model to generate semantically consistent and diverse outputs. This topic-based annotation method enhances the model's understanding of complex scenes, avoids overfitting, and thus improves the accuracy and generalization of generated images. It is suitable for fields such as creative design or virtual reality.

[0020] 6. The settings allow users to select different privacy protection methods (such as encryption algorithms or noise levels) according to their needs, enabling the solution to adapt to diverse privacy requirements and computing environments. This modular design improves the scalability of the method, facilitates integration into existing systems, and reduces deployment costs, providing enterprises with customized solutions.

[0021] 7. Through the continuous training mechanism, the diffusion model can be iteratively optimized using the newly generated biomimetic graph training set, ensuring that the model adapts to changes in data distribution over time. This dynamic optimization capability extends the life cycle of the model, reduces the need for frequent retraining, and lowers long-term maintenance costs.

[0022] 8. By integrating privacy protection technologies such as encrypted computation, differential privacy, and federated learning, end-to-end data security is achieved during the training of the diffusion model, effectively preventing the leakage of user privacy. At the same time, by combining automated preprocessing, multimodal feature extraction, and intelligent labeling processes, training efficiency and data quality are improved, and continuous learning mechanisms are used to optimize model performance, making it highly flexible, strong generalization ability, and low maintenance cost. Overall, it achieves an excellent balance between privacy compliance and the efficiency of artificial intelligence applications. Attached Figure Description

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] Figure 1 This is a flowchart of a diffusion model training method based on privacy computing according to the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of a diffusion model training system based on privacy computing according to the present invention. Detailed Implementation

[0026] The overall approach of the technical solution in this application is as follows: On the client side, privacy computing technology is used to transform the original image data into protected privacy features, ensuring strict protection of the original data privacy at the source. On the server side, a pre-trained diffusion model is used to generate semantically consistent but entirely new biomimetic images based on these privacy features and text prompts, thus transforming the contradiction between privacy protection and data availability into the problem of generating high-quality new data using secure data. Furthermore, an automated model extracts and labels the themes of the text and biomimetic images, efficiently constructing a high-quality biomimetic graph training set, which is ultimately used for continuous optimization of the diffusion model. This achieves the goal of maintaining the model's generation capability while fully protecting data privacy, and realizing a complete technical closed loop from secure data processing to efficient training set construction.

[0027] Please refer to Figures 1 to 2 As shown, a preferred embodiment of the diffusion model training method based on privacy computing of the present invention includes the following steps: Step S1: The client obtains the input image data, text prompts, and settings information, and performs preprocessing on the image data, text prompts, and settings information, including formatting and cleaning. Step S2: Based on the settings information, the client extracts image features from each of the image data through encryption calculation, differential privacy calculation, or federated learning techniques, and encrypts each of the image features into corresponding privacy features. Step S3: The client transmits each of the privacy features and text prompts to the server, and the server stores the received privacy features and text prompts. Step S4: The server inputs the privacy features and text prompts into the pre-trained diffusion model to obtain a biomimetic image; Step S5: Extract the text theme from the text prompt words, extract the biomimetic image theme from the biomimetic image, and label each biomimetic image with prompt word tags based on the text theme and biomimetic image theme to construct a biomimetic image training set. Step S6: Continuously train and optimize the diffusion model using the biomimetic graph training set.

[0028] Through a continuous training mechanism, the diffusion model can be iteratively optimized using newly generated biomimetic graph training sets, ensuring that the model adapts to changes in data distribution over time. This dynamic optimization capability extends the model's lifespan, reduces the need for frequent retraining, and lowers long-term maintenance costs.

[0029] By integrating privacy-preserving technologies such as encrypted computation, differential privacy, and federated learning, end-to-end data security is achieved during the training of diffusion models, effectively preventing the leakage of user privacy. At the same time, by combining automated preprocessing, multimodal feature extraction, and intelligent annotation processes, training efficiency and data quality are improved, and continuous learning mechanisms are used to optimize model performance, making it highly flexible, strong generalization ability, and low maintenance cost. Overall, it achieves an excellent balance between privacy compliance and the efficiency of artificial intelligence applications.

[0030] Step S1 specifically involves: The client acquires input image data, text prompts, and settings information. The settings information carries a privacy protection method, which may be encrypted computation, differential privacy computation, or federated learning. When the privacy protection method is encrypted computation, the settings information also carries the encryption algorithm. When the privacy protection method is differential privacy computation, the settings information also carries the noise level of differential privacy (setting a smaller ε value (noise level) means stronger privacy protection, but may result in a greater loss of data utility). When the privacy protection method is federated learning, the settings information also carries the configuration of federated learning participation (number of local training rounds, the range of local data not leaving the domain, etc.). The image data is formatted using at least the following methods: resizing (adjusting all image data to a uniform resolution (e.g., 224x224 pixels) to fit the model input requirements, which can be achieved through scaling, cropping, or padding); color space conversion (converting image data to a standard color space (e.g., RGB) to avoid problems caused by grayscale or CMYK formats); data normalization (normalizing pixel values ​​to a specific range (e.g., [0,1] or [-1,1]) to improve model training stability); and format conversion (converting image data to a consistent file format (e.g., JPEG, PNG), or directly to tensor format for easier computation). The formatted image data undergoes cleaning processes including at least quality filtering (detecting and removing low-quality image data, such as blurry, overly dark, or overly bright image data, which can be automatically filtered using image quality assessment metrics (such as sharpness and contrast), duplicate detection (using hash algorithms or feature comparison to identify and remove duplicate image data to avoid redundant training data), outlier handling (detecting and removing damaged or invalid image data (such as images with abnormal file sizes or images that cannot be decoded)), and content filtering (filtering out image data containing irrelevant or sensitive content (such as images containing faces or private information) based on business needs to comply with privacy policies), thereby completing the preprocessing of the image data. The text prompts are processed by at least the following methods: text normalization (converting text prompts to lowercase (or uppercase), removing extra spaces, punctuation marks, and special characters to ensure consistency) and tokenization (splitting text prompts into words or tokens using a standard tokenizer such as BERT). The preprocessing of each text prompt word is completed by a tokenizer (adapted to a pre-trained model), encoding conversion (converting text prompt words into numerical sequences or embeddings, such as using a bag-of-words model or a pre-trained language model (such as CLIP's text encoder)), and length normalization (truncating or padding text sequences to a fixed length to ensure consistent model input dimensions). The formatted text prompt words are then cleaned by at least the following steps: spell correction (using spell checking tools (such as Aspell or deep learning-based correctors) to correct common spelling errors), stop word removal (removing common but meaningless words (such as "the", "a"), but ensuring that key descriptive words are retained in the prompt words), irrelevant content filtering (filtering out text that does not match the image content (such as advertising slogans, garbled text), or filtering relevant prompt words based on keywords), and language recognition (ensuring that the text prompt words are in a supported language and translating or converting them). The configuration information is formatted by at least the following steps: structure parsing (parsing the configuration information from its original format (such as JSON, YAML, or command-line parameters) into structured data for easy program access), parameter standardization (converting parameters to a consistent data type and unit (such as converting the noise level ε to a floating-point number)), and default value filling (setting default values ​​for missing parameters (such as the default number of local training epochs being 10 in federated learning configuration)). The formatted configuration information is then cleaned by at least the following steps: validity verification (checking whether privacy protection methods are supported (such as only allowing "encrypted computation", "differential privacy computation", or "federated learning"), and verifying parameter ranges (such as noise level ε being greater than 0)), security checks (ensuring the configuration information does not contain malicious code or injection attacks, such as escaping or filtering input), and consistency checks (verifying the logical relationships between parameters (such as "local data does not leave the domain" being compatible with the number of training epochs in federated learning configuration)). This completes the preprocessing of the configuration information.

[0031] The configuration information allows users to choose different privacy protection methods (such as encryption algorithms or noise levels) according to their needs, enabling the solution to adapt to diverse privacy requirements and computing environments. This modular design improves the scalability of the approach, facilitates integration into existing systems, and reduces deployment costs, providing enterprises with customized solutions.

[0032] Step S2 specifically involves: Based on the privacy protection method carried in the settings information, the client extracts image features from each of the image data in real time through encrypted computation, differential privacy computation, or federated learning techniques, and encrypts each of the image features into corresponding privacy features through homomorphic encryption algorithms.

[0033] By integrating encrypted computing, differential privacy computing, or federated learning techniques, privacy features are extracted and encrypted from image data during the training of the diffusion model. This ensures that the original user data is not exposed to the server in plaintext. This method effectively prevents the risk of privacy leakage, complies with increasingly stringent data protection regulations (such as GDPR), and maintains user trust. It is particularly suitable for data processing scenarios in sensitive fields such as healthcare and finance.

[0034] Step S3 specifically involves: The client transmits each privacy feature and text prompt to the server through a secure channel. The server receives each privacy feature and text prompt in real time and stores each privacy feature and text prompt in the hardware security module.

[0035] By employing secure channels and hardware security modules in the data transmission and storage stages, combined with formatting and cleaning in the preprocessing steps, the system ensures that image and text data are protected from theft or tampering during transmission and eliminates interference from low-quality or abnormal data. This end-to-end security design reduces the risk of data contamination, improves the reliability of training data, and lays a solid foundation for model optimization.

[0036] Step S4 specifically involves: The server inputs the privacy features and text prompts into a pre-trained diffusion model. The diffusion model performs inference using hardware acceleration technology to obtain a biomimetic image that is semantically consistent with the image data corresponding to the privacy features and does not contain user privacy information. Step S5 specifically involves: The text topics are extracted from the text prompts using a pre-trained KeyBERT model, and the biomimetic image topics are extracted from the biomimetic images using a pre-trained BLIP-2 model. Based on the text topics and biomimetic image topics, the prompts are automatically labeled for each biomimetic image using a pre-trained multimodal large model to construct a biomimetic image training set.

[0037] By automating the preprocessing, feature extraction, and annotation processes, the need for manual intervention is reduced. For example, KeyBERT and BLIP-2 models are used for topic extraction and automatic annotation. This not only accelerates the construction of the biomimetic graph training set, but also improves the inference speed of the diffusion model through hardware acceleration technology, thus shortening the overall model training cycle and making it suitable for large-scale real-time applications.

[0038] By leveraging multimodal ensemble, text prompts are combined with image features, and a high-quality training set is constructed using biomimetic images. This enables the diffusion model to generate semantically consistent and diverse outputs. This topic-based annotation method enhances the model's understanding of complex scenes, avoids overfitting, and thus improves the accuracy and generalization of generated images. It is suitable for fields such as creative design or virtual reality.

[0039] A preferred embodiment of the diffusion model training system based on privacy computing of the present invention includes the following modules: The data input preprocessing module is used by the client to obtain input image data, text prompts, and setting information, and to perform preprocessing on the image data, text prompts, and setting information, including formatting and cleaning. The privacy feature extraction module is used by the client to extract image features from each of the image data based on the setting information through encryption calculation, differential privacy calculation or federated learning technology, and encrypt each of the image features into corresponding privacy features; The privacy feature upload module is used by the client to transmit each privacy feature and text prompt to the server, and the server stores the received privacy features and text prompt. A biomimetic image generation module is used by the server to input the privacy features and text prompts into a pre-trained diffusion model to obtain a biomimetic image; The biomimetic image training set construction module is used to extract text topics from the text prompts, extract biomimetic image topics from the biomimetic images, and label each biomimetic image with prompts based on the text topics and biomimetic image topics in order to construct a biomimetic image training set. The diffusion model training module is used to continuously train and optimize the diffusion model using the biomimetic graph training set.

[0040] Through a continuous training mechanism, the diffusion model can be iteratively optimized using newly generated biomimetic graph training sets, ensuring that the model adapts to changes in data distribution over time. This dynamic optimization capability extends the model's lifespan, reduces the need for frequent retraining, and lowers long-term maintenance costs.

[0041] By integrating privacy-preserving technologies such as encrypted computation, differential privacy, and federated learning, end-to-end data security is achieved during the training of diffusion models, effectively preventing the leakage of user privacy. At the same time, by combining automated preprocessing, multimodal feature extraction, and intelligent annotation processes, training efficiency and data quality are improved, and continuous learning mechanisms are used to optimize model performance, making it highly flexible, strong generalization ability, and low maintenance cost. Overall, it achieves an excellent balance between privacy compliance and the efficiency of artificial intelligence applications.

[0042] The data input preprocessing module is specifically used for: The client acquires input image data, text prompts, and settings information. The settings information carries a privacy protection method, which may be encrypted computation, differential privacy computation, or federated learning. When the privacy protection method is encrypted computation, the settings information also carries the encryption algorithm. When the privacy protection method is differential privacy computation, the settings information also carries the noise level of differential privacy (setting a smaller ε value (noise level) means stronger privacy protection, but may result in a greater loss of data utility). When the privacy protection method is federated learning, the settings information also carries the configuration of federated learning participation (number of local training rounds, the range of local data not leaving the domain, etc.). The image data is formatted using at least the following methods: resizing (adjusting all image data to a uniform resolution (e.g., 224x224 pixels) to fit the model input requirements, which can be achieved through scaling, cropping, or padding); color space conversion (converting image data to a standard color space (e.g., RGB) to avoid problems caused by grayscale or CMYK formats); data normalization (normalizing pixel values ​​to a specific range (e.g., [0,1] or [-1,1]) to improve model training stability); and format conversion (converting image data to a consistent file format (e.g., JPEG, PNG), or directly to tensor format for easier computation). The formatted image data undergoes cleaning processes including at least quality filtering (detecting and removing low-quality image data, such as blurry, overly dark, or overly bright image data, which can be automatically filtered using image quality assessment metrics (such as sharpness and contrast), duplicate detection (using hash algorithms or feature comparison to identify and remove duplicate image data to avoid redundant training data), outlier handling (detecting and removing damaged or invalid image data (such as images with abnormal file sizes or images that cannot be decoded)), and content filtering (filtering out image data containing irrelevant or sensitive content (such as images containing faces or private information) based on business needs to comply with privacy policies), thereby completing the preprocessing of the image data. The text prompts are processed by at least the following methods: text normalization (converting text prompts to lowercase (or uppercase), removing extra spaces, punctuation marks, and special characters to ensure consistency) and tokenization (splitting text prompts into words or tokens using a standard tokenizer such as BERT). The preprocessing of each text prompt word is completed by a tokenizer (adapted to a pre-trained model), encoding conversion (converting text prompt words into numerical sequences or embeddings, such as using a bag-of-words model or a pre-trained language model (such as CLIP's text encoder)), and length normalization (truncating or padding text sequences to a fixed length to ensure consistent model input dimensions). The formatted text prompt words are then cleaned by at least the following steps: spell correction (using spell checking tools (such as Aspell or deep learning-based correctors) to correct common spelling errors), stop word removal (removing common but meaningless words (such as "the", "a"), but ensuring that key descriptive words are retained in the prompt words), irrelevant content filtering (filtering out text that does not match the image content (such as advertising slogans, garbled text), or filtering relevant prompt words based on keywords), and language recognition (ensuring that the text prompt words are in a supported language and translating or converting them). The configuration information is formatted by at least the following steps: structure parsing (parsing the configuration information from its original format (such as JSON, YAML, or command-line parameters) into structured data for easy program access), parameter standardization (converting parameters to a consistent data type and unit (such as converting the noise level ε to a floating-point number)), and default value filling (setting default values ​​for missing parameters (such as setting the local training epochs to 10 by default in federated learning configuration)). The formatted configuration information is then cleaned by at least the following steps: validity verification (checking whether privacy protection methods are supported (such as only allowing "encrypted computation", "differential privacy computation", or "federated learning"), and verifying parameter ranges (such as ensuring that the noise level ε must be greater than 0)), security checks (ensuring that the configuration information does not contain malicious code or injection attacks, such as escaping or filtering input), and consistency checks (verifying the logical relationships between parameters (such as ensuring that "local data does not leave the domain" is compatible with the number of training epochs in federated learning configuration)). This completes the preprocessing of the configuration information.

[0043] The configuration information allows users to choose different privacy protection methods (such as encryption algorithms or noise levels) according to their needs, enabling the solution to adapt to diverse privacy requirements and computing environments. This modular design improves the scalability of the approach, facilitates integration into existing systems, and reduces deployment costs, providing enterprises with customized solutions.

[0044] The privacy feature extraction module is specifically used for: Based on the privacy protection method carried in the settings information, the client extracts image features from each of the image data in real time through encrypted computation, differential privacy computation, or federated learning techniques, and encrypts each of the image features into corresponding privacy features through homomorphic encryption algorithms.

[0045] By integrating encrypted computing, differential privacy computing, or federated learning techniques, privacy features are extracted and encrypted from image data during the training of the diffusion model. This ensures that the original user data is not exposed to the server in plaintext. This method effectively prevents the risk of privacy leakage, complies with increasingly stringent data protection regulations (such as GDPR), and maintains user trust. It is particularly suitable for data processing scenarios in sensitive fields such as healthcare and finance.

[0046] The privacy feature uploading module is specifically used for: The client transmits each privacy feature and text prompt to the server through a secure channel. The server receives each privacy feature and text prompt in real time and stores each privacy feature and text prompt in the hardware security module.

[0047] By employing secure channels and hardware security modules in the data transmission and storage stages, combined with formatting and cleaning in the preprocessing steps, the system ensures that image and text data are protected from theft or tampering during transmission and eliminates interference from low-quality or abnormal data. This end-to-end security design reduces the risk of data contamination, improves the reliability of training data, and lays a solid foundation for model optimization.

[0048] The biomimetic image generation module is specifically used for: The server inputs the privacy features and text prompts into a pre-trained diffusion model. The diffusion model performs inference using hardware acceleration technology to obtain a biomimetic image that is semantically consistent with the image data corresponding to the privacy features and does not contain user privacy information. The biomimetic graph training set construction module is specifically used for: The text topics are extracted from the text prompts using a pre-trained KeyBERT model, and the biomimetic image topics are extracted from the biomimetic images using a pre-trained BLIP-2 model. Based on the text topics and biomimetic image topics, the prompts are automatically labeled for each biomimetic image using a pre-trained multimodal large model to construct a biomimetic image training set.

[0049] By automating the preprocessing, feature extraction, and annotation processes, the need for manual intervention is reduced. For example, KeyBERT and BLIP-2 models are used for topic extraction and automatic annotation. This not only accelerates the construction of the biomimetic graph training set, but also improves the inference speed of the diffusion model through hardware acceleration technology, thus shortening the overall model training cycle and making it suitable for large-scale real-time applications.

[0050] By leveraging multimodal ensemble, text prompts are combined with image features, and a high-quality training set is constructed using biomimetic images. This enables the diffusion model to generate semantically consistent and diverse outputs. This topic-based annotation method enhances the model's understanding of complex scenes, avoids overfitting, and thus improves the accuracy and generalization of generated images. It is suitable for fields such as creative design or virtual reality.

[0051] In summary, the advantages of this invention are as follows: 1. The system acquires input image data, text prompts, and settings information from the client. Preprocessing of these data, including formatting and cleaning, is performed. Then, based on the settings information, image features are extracted from each image data using encryption computation, differential privacy computation, or federated learning techniques. Each image feature is encrypted into a corresponding privacy feature, and these privacy features, along with the text prompts, are transmitted to the server. The server stores the received privacy features and text prompts. Next, the server inputs these privacy features and text prompts into a pre-trained diffusion model to obtain biomimetic images. Text topics are extracted from the text prompts, and biomimetic image topics are extracted from the biomimetic images. Based on the text topics and biomimetic image topics, prompt labels are applied to each biomimetic image to construct a biomimetic image training set. Finally, the diffusion model is trained using the biomimetic image training set. The diffusion model undergoes continuous training and optimization. Specifically, on the client side, privacy-preserving computation techniques (such as cryptographic computation, differential privacy, federated learning, and homomorphic encryption algorithms) are used to transform raw image data into protected privacy features, ensuring strict protection of the original data privacy at the source. On the server side, a pre-trained diffusion model is used to generate semantically consistent but entirely new biomimetic images based on these privacy features and text prompts. This transforms the conflict between privacy protection and data usability into the problem of generating high-quality new data using secure data. Furthermore, an automated model extracts and labels the themes of the text and biomimetic images, efficiently constructing a high-quality biomimetic graph training set, which is ultimately used for continuous optimization of the diffusion model. This achieves the goal of maintaining model generation capabilities while fully protecting data privacy, and realizing a complete technical loop from secure data processing to efficient training set construction.

[0052] 2. By integrating encrypted computing, differential privacy computing, or federated learning techniques, privacy features are extracted and encrypted from image data during the training of the diffusion model. This ensures that the original user data is not exposed to the server in plaintext. This method effectively prevents the risk of privacy leakage, complies with increasingly stringent data protection regulations (such as GDPR), and maintains user trust. It is particularly suitable for data processing scenarios in sensitive fields such as healthcare and finance.

[0053] 3. By employing secure channels and hardware security modules in the data transmission and storage stages, combined with formatting and cleaning in the preprocessing steps, we ensured that image and text data were protected from theft or tampering during transmission and eliminated interference from low-quality or abnormal data. This end-to-end security design reduced the risk of data contamination, improved the reliability of training data, and laid a solid foundation for model optimization.

[0054] 4. By automating the preprocessing, feature extraction, and annotation processes, the need for manual intervention is reduced. For example, KeyBERT and BLIP-2 models are used for topic extraction and automatic annotation. This not only accelerates the construction of the biomimetic graph training set, but also improves the inference speed of the diffusion model through hardware acceleration technology, thus shortening the overall model training cycle and making it suitable for large-scale real-time applications.

[0055] 5. By utilizing multimodal ensemble, text prompts are combined with image features, and a high-quality training set is constructed through biomimetic images. This enables the diffusion model to generate semantically consistent and diverse outputs. This topic-based annotation method enhances the model's understanding of complex scenes, avoids overfitting, and thus improves the accuracy and generalization of generated images. It is suitable for fields such as creative design or virtual reality.

[0056] 6. The settings allow users to select different privacy protection methods (such as encryption algorithms or noise levels) according to their needs, enabling the solution to adapt to diverse privacy requirements and computing environments. This modular design improves the scalability of the method, facilitates integration into existing systems, and reduces deployment costs, providing enterprises with customized solutions.

[0057] 7. Through the continuous training mechanism, the diffusion model can be iteratively optimized using the newly generated biomimetic graph training set, ensuring that the model adapts to changes in data distribution over time. This dynamic optimization capability extends the life cycle of the model, reduces the need for frequent retraining, and lowers long-term maintenance costs.

[0058] 8. By integrating privacy protection technologies such as encrypted computation, differential privacy, and federated learning, end-to-end data security is achieved during the training of the diffusion model, effectively preventing the leakage of user privacy. At the same time, by combining automated preprocessing, multimodal feature extraction, and intelligent labeling processes, training efficiency and data quality are improved, and continuous learning mechanisms are used to optimize model performance, making it highly flexible, strong generalization ability, and low maintenance cost. Overall, it achieves an excellent balance between privacy compliance and the efficiency of artificial intelligence applications.

[0059] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A diffusion model training method based on privacy computing, characterized in that: Includes the following steps: Step S1: The client obtains the input image data, text prompts, and settings information, and performs preprocessing on the image data, text prompts, and settings information, including formatting and cleaning. Step S2: Based on the settings information, the client extracts image features from each of the image data through encryption calculation, differential privacy calculation, or federated learning techniques, and encrypts each of the image features into corresponding privacy features. Step S3: The client transmits each of the privacy features and text prompts to the server, and the server stores the received privacy features and text prompts. Step S4: The server inputs the privacy features and text prompts into the pre-trained diffusion model to obtain a biomimetic image; Step S5: Extract the text theme from the text prompt words, extract the biomimetic image theme from the biomimetic image, and label each biomimetic image with prompt word tags based on the text theme and biomimetic image theme to construct a biomimetic image training set. Step S6: Continuously train and optimize the diffusion model using the biomimetic graph training set.

2. The diffusion model training method based on privacy computation as described in claim 1, characterized in that: Step S1 specifically involves: The client obtains the input image data, text prompts, and settings information; the settings information carries a privacy protection method, which is encrypted computation, differential privacy computation, or federated learning technology. When the privacy protection method is encrypted computation, the setting information also carries the encryption algorithm; when the privacy protection method is differential privacy computation, the setting information also carries the noise level of differential privacy; when the privacy protection method is federated learning technology, the setting information also carries the participation configuration of federated learning. The image data is formatted by at least the following processes: size adjustment, color space conversion, data normalization, and format conversion. The formatted image data is then cleaned by at least the following processes: quality filtering, duplicate detection, outlier handling, and content filtering, thereby completing the preprocessing of the image data. The text prompts are formatted by at least text normalization, word segmentation, encoding conversion and length standardization. The formatted text prompts are then cleaned by at least spell correction, stop word removal, irrelevant content filtering and language recognition, thereby completing the preprocessing of the text prompts. The settings information is formatted by at least structural parsing, parameter standardization, and default value filling. The formatted settings information is then cleaned by at least validity verification, security checks, and consistency checks, thereby completing the preprocessing of the settings information.

3. The diffusion model training method based on privacy computation as described in claim 1, characterized in that: Step S2 specifically involves: Based on the privacy protection method carried in the settings information, the client extracts image features from each of the image data in real time through encrypted computation, differential privacy computation, or federated learning techniques, and encrypts each of the image features into corresponding privacy features through homomorphic encryption algorithms.

4. The diffusion model training method based on privacy computation as described in claim 1, characterized in that: Step S3 specifically involves: The client transmits each privacy feature and text prompt to the server through a secure channel. The server receives each privacy feature and text prompt in real time and stores each privacy feature and text prompt in the hardware security module.

5. The diffusion model training method based on privacy computation as described in claim 1, characterized in that: Step S4 specifically involves: The server inputs the privacy features and text prompts into a pre-trained diffusion model. The diffusion model performs inference using hardware acceleration technology to obtain a biomimetic image that is semantically consistent with the image data corresponding to the privacy features and does not contain user privacy information. Step S5 specifically involves: The text topics are extracted from the text prompts using a pre-trained KeyBERT model, and the biomimetic image topics are extracted from the biomimetic images using a pre-trained BLIP-2 model. Based on the text topics and biomimetic image topics, the prompts are automatically labeled for each biomimetic image using a pre-trained multimodal large model to construct a biomimetic image training set.

6. A diffusion model training system based on privacy computing, characterized in that: Includes the following modules: The data input preprocessing module is used by the client to obtain input image data, text prompts, and setting information, and to perform preprocessing on the image data, text prompts, and setting information, including formatting and cleaning. The privacy feature extraction module is used by the client to extract image features from each of the image data based on the setting information through encryption calculation, differential privacy calculation or federated learning technology, and encrypt each of the image features into corresponding privacy features; The privacy feature upload module is used by the client to transmit each privacy feature and text prompt to the server, and the server stores the received privacy features and text prompt. A biomimetic image generation module is used by the server to input the privacy features and text prompts into a pre-trained diffusion model to obtain a biomimetic image; The biomimetic image training set construction module is used to extract text topics from the text prompts, extract biomimetic image topics from the biomimetic images, and label each biomimetic image with prompts based on the text topics and biomimetic image topics in order to construct a biomimetic image training set. The diffusion model training module is used to continuously train and optimize the diffusion model using the biomimetic graph training set.

7. The diffusion model training system based on privacy computation as described in claim 6, characterized in that: The data input preprocessing module is specifically used for: The client obtains the input image data, text prompts, and settings information; the settings information carries a privacy protection method, which is encrypted computation, differential privacy computation, or federated learning technology. When the privacy protection method is encrypted computation, the setting information also carries the encryption algorithm; when the privacy protection method is differential privacy computation, the setting information also carries the noise level of differential privacy; when the privacy protection method is federated learning technology, the setting information also carries the participation configuration of federated learning. The image data is formatted by at least the following processes: size adjustment, color space conversion, data normalization, and format conversion. The formatted image data is then cleaned by at least the following processes: quality filtering, duplicate detection, outlier handling, and content filtering, thereby completing the preprocessing of the image data. The text prompts are formatted by at least text normalization, word segmentation, encoding conversion and length standardization. The formatted text prompts are then cleaned by at least spell correction, stop word removal, irrelevant content filtering and language recognition, thereby completing the preprocessing of the text prompts. The settings information is formatted by at least structural parsing, parameter standardization, and default value filling. The formatted settings information is then cleaned by at least validity verification, security checks, and consistency checks, thereby completing the preprocessing of the settings information.

8. The diffusion model training system based on privacy computing as described in claim 6, characterized in that: The privacy feature extraction module is specifically used for: Based on the privacy protection method carried in the settings information, the client extracts image features from each of the image data in real time through encrypted computation, differential privacy computation, or federated learning techniques, and encrypts each of the image features into corresponding privacy features through homomorphic encryption algorithms.

9. The diffusion model training system based on privacy computation as described in claim 6, characterized in that: The privacy feature uploading module is specifically used for: The client transmits each privacy feature and text prompt to the server through a secure channel. The server receives each privacy feature and text prompt in real time and stores each privacy feature and text prompt in the hardware security module.

10. The diffusion model training system based on privacy computing as described in claim 6, characterized in that: The biomimetic image generation module is specifically used for: The server inputs the privacy features and text prompts into a pre-trained diffusion model. The diffusion model performs inference using hardware acceleration technology to obtain a biomimetic image that is semantically consistent with the image data corresponding to the privacy features and does not contain user privacy information. The biomimetic graph training set construction module is specifically used for: The text topics are extracted from the text prompts using a pre-trained KeyBERT model, and the biomimetic image topics are extracted from the biomimetic images using a pre-trained BLIP-2 model. Based on the text topics and biomimetic image topics, the prompts are automatically labeled for each biomimetic image using a pre-trained multimodal large model to construct a biomimetic image training set.

Images