Multi-mode skin disease detection model automatic training system

Abstract

The invention relates to the technical field of skin disease detection, in particular to a multi-mode skin disease detection model automatic training system which comprises a training process design module used for setting a to-be-trained detection model; the training visualization module is used for training a to-be-trained detection model by using the marked information to obtain a trained detection model; the model test module is used for testing the trained detection model to obtain a tested detection model; and the model deployment module is used for deploying the tested detection model. The training process design module allows a user to customize and set training parameters, algorithms and the like, and is flexibly configured to adapt to different business requirements and data characteristics. The flexibility is beneficial for improving the training efficiency and optimizing the model performance. The model deployment module simplifies the conversion process of the model from training to application, so that the trained model can be quickly and conveniently deployed in an actual production environment.

Description

Technical Field

[0001] This invention relates to the field of dermatology detection technology, and in particular to an automatic training system for a multimodal dermatology detection model. Background Technology

[0002] In today's ever-increasing healthcare landscape, skin diseases, as a common health problem, require early detection and accurate diagnosis for a vital impact on patients' quality of life. However, due to the wide variety of skin diseases and their complex symptoms, traditional diagnostic methods often rely on doctors' personal experience and visual observation. This not only increases the subjectivity of diagnosis but also limits its efficiency and accuracy.

[0003] With the rapid development of artificial intelligence and big data technologies, multimodal data analysis and machine learning algorithms are showing great potential in the medical field. Multimodal data, including skin images, medical record texts, and physiological parameters, provides a rich source of information for the comprehensive analysis of skin diseases. By processing and analyzing this multimodal data through detection models, the characteristics of skin diseases can be captured more accurately, improving the accuracy and efficiency of diagnosis. Existing detection models require training, and the training process is relatively simple and inefficient. Summary of the Invention

[0004] (I) Purpose of the Invention

[0005] The purpose of this invention is to provide an automatic training system for a multimodal skin disease detection model that can improve training efficiency.

[0006] (II) Technical Solution

[0007] To address the above problems, this invention provides an automatic training system for a multimodal dermatology detection model, comprising:

[0008] The data anonymization module is used to receive uploaded information and anonymize the personal information in the uploaded information to obtain the anonymized information;

[0009] The annotation module is used to annotate the desensitized information to obtain annotated information;

[0010] The training process design module is used to set up the detection model to be trained;

[0011] The training visualization module is used to train the detection model to be trained using the labeled information, so as to obtain the trained detection model.

[0012] The model testing module is used to test the trained detection model and obtain the tested detection model.

[0013] The model deployment module is used to deploy the detection model after testing.

[0014] In another aspect of the present invention, preferably, the data desensitization module includes a face desensitization submodule and a text desensitization submodule;

[0015] The uploaded information includes image information and text information;

[0016] The face desensitization submodule uses deep learning blurring methods to mask facial information in image information;

[0017] The text desensitization submodule uses deep learning text detection technology to remove personal information from text information;

[0018] The method of using deep learning blurring to mask facial information in an image includes:

[0019] The eye detection method based on Haar features in cv2.CascadeClassifier is used to detect and mask eyes in image information.

[0020] The annotation module uses preset annotation tools to annotate the desensitized information. The preset annotation tools include interactive annotation tools or trained detection models.

[0021] In another aspect of the present invention, preferably, the training process design module selects the detection model to be trained by dragging and dropping on a page, and sets the parameters of the detection model to be trained.

[0022] In another aspect of the present invention, preferably, the training visualization module is used to train the detection model to be trained using labeled information, and to obtain the trained detection model, including:

[0023] The training progress, training logs, training loss, accuracy, and GPU utilization are displayed in real time during the training process.

[0024] In another aspect of the present invention, preferably, the deployment method includes Docker deployment and conda deployment.

[0025] In another aspect of the present invention, preferably, the detection model to be trained includes:

[0026] The image information is processed using a convolutional neural network to extract features and obtain image features.

[0027] The text information is processed using a BERT pre-trained model to extract text features, thus obtaining text features.

[0028] The image features and text features are then concatenated for skin disease detection.

[0029] The convolutional neural network includes multiple convolutional layers and multiple pooling layers, which are connected alternately in sequence.

[0030] Each neuron in the convolutional layer is locally connected to the neurons in the previous convolutional layer;

[0031] Each convolutional layer includes a convolutional kernel, and the weights of each convolutional kernel are shared among all neurons;

[0032] Set a non-linear activation function between the convolutional layer and the pooling layer;

[0033] In another aspect of the present invention, preferably, the detection model to be trained includes: a vgg16 model, a vgg19 model, and a vgg16 network structure model.

[0034] In another aspect of the present invention, preferably, the detection model is trained using federated learning based on training data from multiple participants, and the method further includes the step of determining the training data from the multiple participants:

[0035] The classification rules are issued to the first participant and the second participant. The classification rules are used to classify the data of the first participant and the second participant into corresponding levels. The first participant and the second participant classify the data according to the classification rules to determine the first matching data of the first participant and the second matching data of the second participant. The first matching data and the second matching data are data belonging to the same level or adjacent levels.

[0036] Generate a first key pair containing a first public key and a first private key that conforms to homomorphic encryption rules, and a second key pair containing a second public key and a second private key that conforms to homomorphic encryption rules;

[0037] The first public key and the second public key are issued to the first participant and the second participant. The first participant and the second participant fluctuate the first matching data and the second matching data to obtain the first upper limit value and the first lower limit value of the first matching data, as well as the second upper limit value and the second lower limit value of the second matching data. The first participant uses the first public key to perform homomorphic encryption on the first upper limit value and uses the second public key to perform homomorphic encryption on the first lower limit value. The second participant uses the first public key to perform homomorphic encryption on the first lower limit value and uses the second public key to perform homomorphic encryption on the first upper limit value.

[0038] Receive the first upper limit value encrypted by the first participant and the second lower limit value encrypted by the second participant, compare their sizes in the encrypted state, and decrypt them using the first private key to determine the first comparison result;

[0039] Receive the first lower limit value encrypted by the first participant and the second upper limit value encrypted by the second participant, compare their sizes in the encrypted state, and decrypt them using the second private key to determine the second comparison result;

[0040] Based on the first comparison result and the second comparison result, the similarity analysis result of the first matching data and the second matching data is determined;

[0041] Based on the similarity analysis results, the first matching data of the first participant and the second matching data of the second participant are filtered to determine the training data.

[0042] In another aspect of the present invention, preferably, the step of training the detection model based on training data from multiple participants using a federated learning approach includes:

[0043] The detection model is trained using federated learning based on first training data from multiple participants that conform to a non-independent and identically distributed model.

[0044] The detection model is trained using federated learning based on second training data where the similarity between multiple participants meets a preset threshold.

[0045] in,

[0046] The detection model is trained using federated learning based on initial training data from multiple participants that conform to a non-independent and identically distributed model, including:

[0047] The participant models for the first and second participants are determined based on the global model;

[0048] The participant model is distributed to the first participant and the second participant. The first participant and the second participant train the participant model based on the first training data and upload the obtained local parameters.

[0049] Receive local parameters uploaded by the first and second participants, perform comprehensive analysis, and obtain global parameters;

[0050] Based on the global parameters, the parameter update information for the first and second participants is determined and sent to the first and second participants for the next round of iterative training until the trained global model is obtained.

[0051] The detection model is trained using a federated learning approach based on second training data where the similarity among multiple participants meets a preset threshold. This includes:

[0052] Based on the global model, the participant models of the first and second participants are determined. Based on the similarity analysis results of the first and second participants, a parameter simplification matrix for simplifying the local parameters of the participants and a parameter restoration matrix for restoring the simplified parameters are determined. The parameter simplification matrix includes a first data transformation matrix and a compression ratio, and the parameter restoration matrix includes a second data transformation matrix and a restoration ratio. The parameter simplification matrix is ​​determined based on the level classification rules and the fluctuation range of the participant data. The fluctuation range of the participant data is determined based on the upper and lower limits of the data.

[0053] The participant model and parameter simplification matrix are distributed to the first participant and the second participant. The first participant and the second participant train the participant model based on the second training data and obtain local parameters. After simplifying the local parameters according to the parameter simplification matrix, the simplified parameters are formed and uploaded.

[0054] The system receives simplified parameters uploaded by the first and second participants, and restores the simplified parameters according to the parameter restoration matrix for comprehensive analysis to obtain global parameters.

[0055] Based on the global parameters, the parameter update information for the first and second participants is determined and sent to the first and second participants for the next round of iterative training until the trained global model is obtained.

[0056] In another aspect of the present invention, preferably, the step of restoring the simplified parameters based on the parameter restoration matrix includes:

[0057] The simplified parameters of the first and second participants are combined, and the simplified parameters are restored based on the parameter restoration matrix.

[0058] When the restoration of the first simplified parameter of the target participant fails, a parameter acquisition instruction is sent to the target participant so that the target participant generates a second simplified parameter based on the parameter acquisition instruction and local parameters, and uploads it. The data volume of the second simplified parameter is greater than the data volume of the first simplified parameter.

[0059] Receive the second simplified parameters and restore the parameters. When the parameter restoration is successful, update the parameter simplification matrix and the parameter restoration matrix, and send the updated parameter simplification matrix to the target participant to update the target participant's parameter simplification matrix so that simplified parameters can be generated based on the updated parameter simplification matrix in the next iteration.

[0060] (III) Beneficial Effects

[0061] The above-described technical solution of the present invention has the following beneficial technical effects:

[0062] This invention includes a data anonymization module, a crucial step in ensuring personal privacy and data protection. Before data is used for training or analysis, sensitive personal information is anonymized, effectively preventing privacy leaks, complying with data protection regulations, and enhancing the security and compliance of data use. The annotation module accurately labels the anonymized information, providing high-quality training data for machine learning models. The training process design module allows users to customize and set training parameters, algorithms, etc., flexibly configuring them to adapt to different business needs and data characteristics. This flexibility helps improve training efficiency and optimize model performance. The model testing module comprehensively tests the trained model, verifying its performance and robustness in different scenarios. The model deployment module simplifies the conversion process from training to application, enabling trained models to be quickly and easily deployed to real-world production environments. This helps shorten product development cycles and improve market competitiveness. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the overall structure of one embodiment of the present invention;

[0064] Figure 2 This is a schematic diagram of the overall structure of a training detection model according to an embodiment of the present invention;

[0065] Figure 3 This is an interface diagram of the annotation module according to an embodiment of the present invention;

[0066] Figure 4 This is a diagram of the training process design module interface according to an embodiment of the present invention;

[0067] Figure 5 This is a schematic diagram of a convolutional neural network structure according to an embodiment of the present invention;

[0068] Figure 6 This is a schematic diagram of the BERT pre-trained model structure according to an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0070] The accompanying drawings illustrate a layer structure according to an embodiment of the present invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0071] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0072] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0073] The invention will now be described in more detail with reference to the accompanying drawings. In the various drawings, the same elements are indicated by similar reference numerals. For clarity, the various parts in the drawings are not drawn to scale.

[0074] Example 1

[0075] An automatic training system for a multimodal skin disease detection model. Figure 1 A schematic diagram of the overall structure of an embodiment of the present invention is shown, as follows. Figure 1 As shown, it includes:

[0076] The data anonymization module receives uploaded information and anonymizes the personal information within it, obtaining the anonymized information. Anonymization protects personal information and prevents its leakage. The uploaded information is used for training and includes image and text information. The text information may include private information such as ID card details, name, and home address, while the image information may include facial features such as eyes. Anonymizing personal information facilitates patient privacy protection, complies with data protection regulations, and enhances the security and compliance of data use.

[0077] In this embodiment, the data desensitization module includes a face desensitization submodule and a text desensitization submodule. The face desensitization submodule uses a deep learning blurring method to mask facial information in the image information. Further, masking facial information in the image information using a deep learning blurring method includes: using the Haar feature-based eye detection method in cv2.CascadeClassifier to detect and mask eyes in the image information. cv2.CascadeClassifier uses a pre-trained Haar feature classifier to detect specific objects (such as eyes) in the image. These features are designed based on the grayscale variation patterns of local image regions, and have good detection effects for regions with obvious feature differences, such as eyes. Besides using traditional computer vision libraries (such as cv2.CascadeClassifier in OpenCV) for simple eye detection, the face desensitization submodule can integrate more advanced deep learning models, such as Faster R-CNN, YOLO (You Only Look Once), or MTCNN (Multi-task Cascaded Convolutional Networks), to achieve more accurate detection and localization of faces and their key features (such as eyes, nose, mouth, etc.). Once a face or key features are detected, the system can employ various blurring strategies, such as Gaussian blur, median blur, or pixelation blur, to mask the detected area. The degree of blurring can be adjusted according to privacy protection needs to ensure that useful image information for dermatological diagnosis is preserved as much as possible while protecting privacy. To improve the desensitization effect, the system can also dynamically adjust the shape and size of the blurred area based on the size, angle, and position of the face, ensuring full coverage without excessively blurring non-privacy areas.

[0078] The text anonymization submodule utilizes deep learning text detection technology to remove personal information from text. This submodule can employ deep learning algorithms based on BERT (Bidirectional Encoder Representations from Transformers) or other pre-trained language models to perform Named Entity Recognition (NER) on the text. These models, trained on extensive text data, can accurately identify personal information (such as names, ID numbers, and addresses) within the text. To improve recognition accuracy in specific domains (such as the medical field), the system can build and integrate a custom dictionary containing common medical terms and privacy information vocabulary. For detected personal information, the system can directly remove it from the text, retaining only non-privacy information related to dermatology diagnosis. To avoid affecting the overall meaning of the text due to missing information, the system can also employ a replacement strategy, replacing detected personal information with placeholders or forged random information to ensure the readability and integrity of the text.

[0079] The annotation module is used to annotate the desensitized information to obtain annotated information; Figure 3 An interface diagram of the annotation module according to an embodiment of the present invention is shown; as follows: Figure 3 As shown, the annotation module uses preset annotation tools to annotate the desensitized information. These preset annotation tools include interactive annotation tools or trained detection models. Figure 3 As shown, users can easily complete data annotation through simple drag-and-drop and click operations. The annotation module provides an intuitive graphical user interface, allowing users to directly draw annotation boxes, polygons, lines, etc., on the image to mark skin disease areas, lesion features, or other key information through simple drag-and-drop, click, and selection operations. The tool also provides real-time feedback and preview functions to help users better monitor the annotation progress and results. Furthermore, this implementation provides a pre-annotation method, which can load pre-trained models for pre-annotation and incremental training. Pre-annotation function: Utilizing a pre-trained skin disease detection model to perform preliminary annotation on desensitized images, serving as a reference or starting point for user annotation. This pre-annotation method can significantly reduce the workload of manual annotation and improve the accuracy and consistency of annotation. After manual correction and supplementation of the pre-annotation results, this newly annotated data can be used for incremental training of the original model, further improving the model's detection performance. This iterative annotation and training process helps to continuously optimize the model.

[0080] The training process design module is used to set up the detection model to be trained; Figure 4 The diagram shows the interface of the training process design module according to an embodiment of the present invention, such as... Figure 4As shown, the training process design module allows users to select the detection model to be trained via page drag-and-drop and set the parameters of the detection model. Page drag-and-drop operation: as shown... Figure 4 As shown, the training workflow design module allows users to easily select the detection model to be trained through an intuitive graphical interface and drag-and-drop operations. This design lowers the technical barrier, enabling even non-professional users to quickly get started. Model Library Support: The module has a rich built-in model library, including pre-trained deep learning models and classic machine learning algorithms, allowing users to choose suitable models for training based on their actual needs. Custom Models: In addition to selecting built-in models, the module also supports users uploading custom model structures to meet training requirements in specific scenarios. Detailed Parameter Settings: For the selected detection model, users can configure its parameters in detail within the training workflow design module, including key parameters such as learning rate, batch size, number of iterations, and optimizer. Proper setting of these parameters is crucial for the training effect and efficiency of the model. Parameter Saving and Restoration: The module supports saving configured parameters as a configuration file, allowing users to quickly restore or reuse these parameter settings in future training.

[0081] The training visualization module is used to train the detection model using labeled information, resulting in a trained model. It displays the training progress, training logs, training loss, accuracy, and GPU utilization in real time. The module uses progress bars and percentages to show the current training progress, providing users with an intuitive understanding of the training process. Based on the current training speed and remaining training time, the module can also estimate and display the remaining training time, helping users manage their time effectively. The module records detailed log information during the training process, including each step's operation, parameter settings, and output results, facilitating user tracking and problem analysis. In addition to training progress, the module also displays key metrics in real time, such as loss value, accuracy, recall, F1 score, and hardware resource information like GPU utilization. This information is crucial for evaluating training effectiveness and adjusting training strategies.

[0082] In one embodiment of the present invention, the detection model to be trained further includes:

[0083] Image features are obtained by extracting features from image information using a convolutional neural network;

[0084] Text features are extracted from text information using a BERT pre-trained model.

[0085] Skin disease detection is performed by concatenating image features and text features.

[0086] Figure 5A schematic diagram of a convolutional neural network structure according to an embodiment of the present invention is shown, as follows: Figure 5 As shown, a convolutional neural network includes multiple convolutional layers and multiple pooling layers, which are connected alternately in sequence. 1. The convolutional layer is the core component of the convolutional neural network. It uses a set of learnable filters (convolutional kernels) to slide across the input image to extract local features. Each convolutional kernel learns the ability to extract certain specific features (such as edges, textures, etc.).

[0087] Neurons in each convolutional layer are locally connected to neurons in the layer above them; these connections are local rather than fully connected. This locally connected structure mimics the workings of the human visual system, allowing for more efficient extraction of local features.

[0088] Each convolutional layer includes a convolutional kernel, and the weights of each kernel are shared across all neurons. Within a convolutional layer, all neurons use the same kernel, meaning that the same features can be extracted at different locations. This significantly reduces the number of parameters in the network and improves parameter utilization.

[0089] Pooling layers. By downsampling the feature map, the most important features are extracted, reducing the number of parameters and computational complexity, and enhancing the translation invariance of features. Common pooling methods include max pooling and average pooling.

[0090] A non-linear activation function is placed between the convolutional and pooling layers. Between these layers, the network uses a non-linear activation function (such as ReLU) to enhance the expressive power of features, introducing a non-linear factor. This is a crucial guarantee that convolutional networks can learn complex features.

[0091] Figure 6 A schematic diagram of the BERT pre-trained model structure according to an embodiment of the present invention is shown, as follows: Figure 6 As shown, the main function of the BERT pre-trained model is to convert sentences or words into corresponding computer vectors. The input parameters of the model consist of word embeddings, segmentation embeddings, and position encoding tensors. The final embedding vector is the result of directly adding the above three vectors.

[0092] In one embodiment of the present invention, the detection model to be trained further includes: a VGG16 model, a VGG19 model, and a VGG16 network structure model. VGG16 consists of 16 weighted layers, including 13 convolutional layers and 3 fully connected layers. The convolutional layers use small 3x3 convolutional kernels, allowing for the capture of richer features in deeper networks. The VGG16 network stacks multiple small convolutional layers together, replacing a single large convolutional layer. This not only increases network depth and enhances feature extraction capabilities but also results in a relatively small number of parameters. After the convolutional layers, VGG16 uses 2x2 max-pooling layers for downsampling, reducing the number of parameters and computational cost, while also extracting more abstract features. The last three layers of VGG16 are fully connected layers used for classifying the extracted features. The fully connected layers flatten and connect the previous feature maps.

[0093] The model testing module is used to test the trained detection model and obtain a tested detection model. Testing includes two methods: page-based invocation and API invocation. For page-based invocation, users can test the model through the system's graphical user interface (GUI). This method is intuitive and easy to use; users only need to input the image or text to be detected on the interface, and the system can automatically invoke the trained model to make predictions and display the results. Page-based invocation testing helps non-technical users quickly verify the model's effectiveness and also facilitates debugging and optimization by technical personnel. API invocation testing is essential for detection models that need to be integrated into other systems or applications. Through clearly defined API interfaces, other systems can invoke the model for predictions via HTTP requests, etc. API invocation testing needs to ensure the stability and response speed of the interface, and also requires strict validation and error handling of input data to ensure the system's security and reliability.

[0094] The model deployment module is used to deploy the tested detection model. Deployment methods include Docker deployment and conda deployment. Docker is a lightweight containerization technology that can package applications and their dependencies into a single container, running on any Docker-enabled platform. Deploying models using Docker ensures consistency and portability across different environments, while facilitating version control and rollback operations. Docker deployment can also leverage container orchestration tools (such as Kubernetes) for large-scale deployment and management, improving system scalability and stability. Conda is an open-source package, dependency, and environment manager suitable for programming languages ​​such as Python. It helps users quickly install, run, and upgrade packages and their dependencies. Deploying models using Conda ensures precise management of the model's required dependencies, avoiding version conflicts and dependency region issues. Conda deployment is suitable for scenarios requiring fine-grained control over dependencies, such as scientific computing and data analysis.

[0095] Furthermore, in this embodiment, the detection model is trained using federated learning based on training data from multiple participants. Federated learning consists of a central party and multiple participants. During training, the detection model is deployed at the central party as the global model, and the models deployed at each participant are participant models. The participant models can be all or part of the global model. The detection model can be trained on the server side.

[0096] The method also includes the step of determining training data for multiple participants:

[0097] A grading rule is issued to the first and second participants. This grading rule is used to classify the data of the first and second participants into corresponding grades. The first and second participants divide the data according to the grading rule to determine the first matching data of the first participant and the second matching data of the second participant. The first matching data and the second matching data belong to the same grade or adjacent grades. By issuing the grading rule to multiple participants (such as the first and second participants), each participant can independently classify their own data into grades, thereby identifying data belonging to the same grade or adjacent grades as matching data. For example, data such as 9.5, 10.6, 11.1, and 11.5 can be classified into integer grades 9, 10, 11, and 11. Data belonging to the same integer grade are similar. In addition, data in adjacent grades may also be similar. Therefore, similarity matching can be performed on adjacent integer grades later. The grading rule can be formulated according to specific application scenarios and data characteristics, such as based on data distribution, statistical characteristics, or business logic.

[0098] After determining the first matching data of the first participant and the second matching data of the second participant, a first key pair containing a first public key and a first private key that conforms to the homomorphic encryption rule is generated, and a second key pair containing a second public key and a second private key that conforms to the homomorphic encryption rule is generated.

[0099] A first public key and a second public key are issued to a first participant and a second participant. The first and second participants then perform a fluctuation on the first and second matching data to obtain a first upper limit and a first lower limit value for the first matching data, as well as a second upper limit and a second lower limit value for the second matching data. The first participant uses the first public key to homomorphically encrypt the first upper limit value and the second public key to homomorphically encrypt the first lower limit value. Similarly, the second participant uses the first public key to homomorphically encrypt the first lower limit value and the second public key to homomorphically encrypt the first upper limit value. Here, the fluctuation involves transforming or adjusting the form of the first and second matching data to obtain their upper and lower limits. For example, the upper and lower limits of the first and second matching data can be calculated based on their characteristics (such as distribution, range, average value, etc.).

[0100] Receive the first upper limit value encrypted by the first participant and the second lower limit value encrypted by the second participant, compare their sizes in the encrypted state, and decrypt them using the first private key to determine the first comparison result;

[0101] Receive the first lower limit value encrypted by the first participant and the second upper limit value encrypted by the second participant, compare their sizes in the encrypted state, and decrypt them using the second private key to determine the second comparison result;

[0102] Based on the first and second comparison results, the similarity analysis results of the first and second matching data are determined. The first and second comparison results determine whether the two data sets have overlapping parts; if there is overlap, they are similar; otherwise, they are dissimilar. Since the first upper limit value and the second lower limit value are encrypted using the same public key, calculations can be performed in an encrypted state. The difference between the two values ​​can be calculated, and then it can be determined whether the difference is greater than or less than zero. In this embodiment, by distributing public and private keys, each participating party can perform data comparison and similarity analysis while protecting data privacy. Through homomorphic encryption technology, the data of the first and second participating parties remains encrypted throughout the transmission and comparison process, effectively preventing the risk of data leakage. The process of comparing the size of two data sets can be completed locally by the participating party (with minimal computation on the server side), and the result is uploaded to the server for decryption to determine the size comparison result. The process of comparing the size of two data sets can also be completed on the server side. The participating party is the party that owns the data, and the server is the party that performs the data calculations.

[0103] Based on the similarity analysis results, the first matching data of the first participant and the second matching data of the second participant are filtered to determine the training data. For data from multiple participants, the similarity of the data from multiple participants can be analyzed, and the data can be classified according to the similarity. This ensures that the training data is not independently and identically distributed, or similar data from different participants can be used for model training. By processing a portion of the similar data from multiple participants, data conforming to a non-independent and identically distributed distribution can be obtained. Non-independent and identically distributed data is closer to the real data and is not concentrated on certain data points, which can improve the model training effect. This solution can also transform the data into independently and identically distributed data through processing of similar data for training.

[0104] Furthermore, in this embodiment, the training process is divided into model training on dissimilar training data and model training on similar training data. The dissimilar training process is a normal federated learning process. The similar training process can save parameters uploaded by the participants. The steps of training the detection model based on the training data of multiple participants using federated learning include:

[0105] The detection model is trained using federated learning based on first training data from multiple participants that conform to a non-independent and identically distributed model.

[0106] The detection model is trained using federated learning based on second training data where the similarity between multiple participants meets a preset threshold.

[0107] in,

[0108] The detection model is trained using federated learning based on initial training data from multiple participants that conform to a non-independent and identically distributed model, including:

[0109] The participant models for the first and second participants are determined based on the global model; the participant model can be a part or the entirety of the global model.

[0110] The participant model is distributed to the first and second participants. The first and second participants train their models using the first training data and upload the obtained local parameters. Each participant then trains the received participant model using its own first training data (i.e., data that conforms to a non-independent and identically distributed distribution). During training, each participant uploads its trained local parameters (such as weights and biases) to the central unit.

[0111] The system receives local parameters uploaded by the first and second participants and performs comprehensive analysis to obtain global parameters; the central unit receives local parameters uploaded by all participants and updates the global parameters of the global model through comprehensive analysis (such as weighted averaging).

[0112] Based on the global parameters, parameter update information for the first and second participants is determined and distributed to them for the next round of iterative training until a trained global model is obtained. Based on the updated global parameters, new parameter update information is generated for each participant and distributed to them. Participants adjust their models according to this update information and prepare for the next round of iterative training. This process is repeated until the global model achieves the predetermined training effect or reaches the preset number of training iterations.

[0113] The detection model is trained using a federated learning approach based on second training data where the similarity among multiple participants meets a preset threshold. This includes:

[0114] Based on the global model, the participant models for the first and second participants are determined. Then, based on the similarity analysis results between the first and second participants, a parameter simplification matrix is ​​determined to simplify the local parameters of the participants, and a parameter restoration matrix is ​​determined to restore the simplified parameters. When the data similarity between the first and second participants is high, there may be a large number of repetitive or similar parts in the trained local parameters. To save transmission bandwidth and time, parameter simplification and parameter restoration matrices are introduced to optimize this process.

[0115] The parameter simplification matrix is ​​used to simplify the local parameters of the participants. It includes a first data transformation matrix and a compression ratio. The first data transformation matrix maps the local parameters from one space to another for compression; this matrix can be a linear or nonlinear transformation designed based on data characteristics. The compression ratio defines the degree of parameter compression during simplification; a higher compression ratio results in less data transmission. The parameter simplification matrix is ​​determined based on the hierarchical classification rules and the fluctuation range of the participant's data, which is determined by the upper and lower limits of the data. The parameter restoration matrix is ​​used at the central point to restore the simplified parameters to their original or near-original form for comprehensive analysis. It includes a second data transformation matrix and a restoration ratio. The second data transformation matrix, opposite to the first data transformation matrix, maps the simplified parameters back to their original space. The restoration ratio defines the degree of parameter recovery during the restoration process. The parameter simplification matrix is ​​determined based on the hierarchical classification rules and the fluctuation range of the participant's data, which is determined by the upper and lower limits of the data. The data of the first and second participants have high similarity; for example, if parameters 1-100 and 50-150 were originally generated respectively, there may be some overlap. Use a simplified matrix to simplify the parameters uploaded by each participant, such as adjusting the first participant's parameters to 1-80 and the second participant's parameters to 80-150.

[0116] The central operator first generates a participant model based on the global model for each participant. A parameter simplification matrix is ​​calculated based on the similarity and fluctuation range of their data. Then, the participant model and parameter simplification matrix are distributed to the first and second participants. The first and second participants train their participant models using the second training data, obtaining local parameters. These local parameters are then simplified using the parameter simplification matrix to form simplified parameters, which are then uploaded. Each participant then trains its received participant model using its own second training data. The training process involves multiple iterations. In each iteration, the central operator receives the simplified parameters uploaded by the first and second participants and restores them using the parameter restoration matrix for comprehensive analysis to obtain the global parameters.

[0117] Based on the global parameters, the parameter update information for the first and second participants is determined and sent to the first and second participants for the next round of iterative training until a trained global model is obtained.

[0118] Furthermore, in this embodiment, the step of restoring the simplified parameters based on the parameter restoration matrix includes:

[0119] The simplified parameters from the first and second participants are combined, and then restored using the parameter restoration matrix. The central party receives simplified parameters from the first and second participants, which are generated by each participant based on their respective second training data and parameter restoration matrices. The central party combines and restores these simplified parameters using the parameter restoration matrix; the combination process may involve sorting or concatenating simplified parameters from different participants according to certain rules (such as timestamps, data batches, etc.). The restoration process uses the second data transformation matrix and restoration rate in the parameter restoration matrix to map the simplified parameters from the low-dimensional space back to the original high-dimensional space and restore their original accuracy.

[0120] During the restoration process, some simplified parameters may fail to be restored successfully. This may be due to information loss during data conversion, decreased accuracy caused by excessive compression ratio, or mismatch between the parameter simplification matrix and the parameter restoration matrix.

[0121] When the restoration of the first simplified parameters for the target participant fails, a parameter retrieval instruction is sent to the target participant, enabling the target participant to generate and upload second simplified parameters based on the parameter retrieval instruction and local parameters. The data volume of the second simplified parameters is greater than that of the first simplified parameters. When the central party detects that the restoration of the first simplified parameters for the target participant has failed, the central party will send a parameter retrieval instruction to the target participant. This instruction contains information about which parameters need to be re-retrieved. After receiving the parameter retrieval instruction, the target participant will generate second simplified parameters based on the instruction and local parameters. The data volume of the second simplified parameters is usually larger than that of the first simplified parameters because it may contain more information or higher precision to compensate for the information loss during the first simplification process.

[0122] The system receives the second simplified parameters and performs parameter restoration. Upon successful restoration, it updates the parameter simplification matrix and the parameter restoration matrix, and sends the updated parameter simplification matrix to the target participant to update their parameter simplification matrix. This allows the target participant to generate simplified parameters based on the updated parameter simplification matrix in the next iteration. Updating the parameter simplification matrix and the parameter restoration matrix is ​​to prevent further restoration failures.

[0123] This invention includes a data anonymization module, a crucial step in ensuring personal privacy and data protection. Before data is used for training or analysis, sensitive personal information is anonymized, effectively preventing privacy leaks, complying with data protection regulations, and enhancing the security and compliance of data use. The annotation module accurately labels the anonymized information, providing high-quality training data for machine learning models. The training process design module allows users to customize and set training parameters, algorithms, etc., flexibly configuring them to adapt to different business needs and data characteristics. This flexibility helps improve training efficiency and optimize model performance. The model testing module comprehensively tests the trained model, verifying its performance and robustness in different scenarios. The model deployment module simplifies the conversion process from training to application, enabling trained models to be quickly and easily deployed to real-world production environments. This helps shorten product development cycles and improve market competitiveness.

[0124] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

[0125] The above description does not provide detailed explanations of the technical aspects of each layer's patterning and etching. However, those skilled in the art should understand that various methods existing in the prior art can be used to form layers and regions of the desired shape. Furthermore, to form the same structure, those skilled in the art can also design methods that are not entirely identical to those described above.

[0126] The present invention has been described above with reference to embodiments thereof. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

[0127] Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions, and modifications can be made to the embodiments of the present invention without departing from the spirit and scope of the invention.

[0128] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An automatic training system for a multimodal skin disease detection model, characterized in that, include: The data anonymization module is used to receive uploaded information and anonymize the personal information in the uploaded information to obtain the anonymized information; The annotation module is used to annotate the desensitized information to obtain annotated information; The training process design module is used to set up the detection model to be trained; The training visualization module is used to train the detection model to be trained using the labeled information, and obtain the trained detection model. The model testing module is used to test the trained detection model and obtain the tested detection model. The model deployment module is used to deploy the detection model after testing.

2. The system according to claim 1, characterized in that: The data desensitization module includes a face desensitization submodule and a text desensitization submodule; The uploaded information includes image information and text information; The face desensitization submodule uses deep learning blurring methods to mask facial information in image information; The text desensitization submodule uses deep learning text detection technology to remove personal information from text information; The method of using deep learning blurring to mask facial information in an image includes: The eye detection method based on Haar features in cv2.CascadeClassifier is used to detect and mask eyes in image information. The annotation module uses preset annotation tools to annotate the desensitized information. The preset annotation tools include interactive annotation tools or trained detection models.

3. The system according to claim 1, characterized in that: The training process design module allows users to select a detection model to be trained by dragging and dropping on a page, and then set the parameters of the detection model to be trained.

4. The system according to claim 1, characterized in that: The training visualization module is used to train the detection model to be trained using the labeled information, and to obtain the trained detection model, including: The training progress, training logs, training loss, accuracy, and GPU utilization are displayed in real time during the training process.

5. The system according to claim 1, characterized in that: The deployment methods include Docker deployment and conda deployment.

6. The system according to claim 2, characterized in that: The detection model to be trained includes: The image information is processed using a convolutional neural network to extract features and obtain image features. The text information is processed using a BERT pre-trained model to extract text features, thus obtaining text features. The image features and text features are then concatenated for skin disease detection. The convolutional neural network includes multiple convolutional layers and multiple pooling layers, which are connected alternately in sequence. Each neuron in the convolutional layer is locally connected to the neurons in the previous convolutional layer; Each convolutional layer includes a convolutional kernel, and the weights of each convolutional kernel are shared among all neurons; Set a non-linear activation function between the convolutional layer and the pooling layer.

7. The system according to claim 6, characterized in that: The detection models to be trained include: the vgg16 model, the vgg19 model, and the vgg16 network structure model.

8. The system according to claim 1, characterized in that: The detection model is trained using federated learning based on training data from multiple participants. The method also includes the step of determining the training data from the multiple participants. The classification rules are issued to the first participant and the second participant. The classification rules are used to classify the data of the first participant and the second participant into corresponding levels. The first participant and the second participant classify the data according to the classification rules to determine the first matching data of the first participant and the second matching data of the second participant. The first matching data and the second matching data are data belonging to the same level or adjacent levels. Generate a first key pair containing a first public key and a first private key that conforms to homomorphic encryption rules, and a second key pair containing a second public key and a second private key that conforms to homomorphic encryption rules; The first public key and the second public key are issued to the first participant and the second participant. The first participant and the second participant fluctuate the first matching data and the second matching data to obtain the first upper limit value and the first lower limit value of the first matching data, as well as the second upper limit value and the second lower limit value of the second matching data. The first participant uses the first public key to perform homomorphic encryption on the first upper limit value and uses the second public key to perform homomorphic encryption on the first lower limit value. The second participant uses the first public key to perform homomorphic encryption on the first lower limit value and uses the second public key to perform homomorphic encryption on the first upper limit value. Receive the first upper limit value encrypted by the first participant and the second lower limit value encrypted by the second participant, compare their sizes in the encrypted state, and decrypt them using the first private key to determine the first comparison result; Receive the first lower limit value encrypted by the first participant and the second upper limit value encrypted by the second participant, compare their sizes in the encrypted state, and decrypt them using the second private key to determine the second comparison result; Based on the first comparison result and the second comparison result, the similarity analysis result of the first matching data and the second matching data is determined; Based on the similarity analysis results, the first matching data of the first participant and the second matching data of the second participant are filtered to determine the training data.

9. The system according to claim 8, characterized in that: The steps for training a detection model using federated learning based on training data from multiple participants include: The detection model is trained using federated learning based on first training data from multiple participants that conform to a non-independent and identically distributed model. The detection model is trained using federated learning based on second training data where the similarity between multiple participants meets a preset threshold. in, The detection model is trained using federated learning based on initial training data from multiple participants that conform to a non-independent and identically distributed model, including: The participant models for the first and second participants are determined based on the global model; The participant model is distributed to the first participant and the second participant. The first participant and the second participant train the participant model based on the first training data and upload the obtained local parameters. Receive local parameters uploaded by the first and second participants, perform comprehensive analysis, and obtain global parameters; Based on the global parameters, the parameter update information for the first and second participants is determined and sent to the first and second participants for the next round of iterative training until the trained global model is obtained. The detection model is trained using a federated learning approach based on second training data where the similarity among multiple participants meets a preset threshold. This includes: Based on the global model, the participant models of the first and second participants are determined. Based on the similarity analysis results of the first and second participants, a parameter simplification matrix for simplifying the local parameters of the participants and a parameter restoration matrix for restoring the simplified parameters are determined. The parameter simplification matrix includes a first data transformation matrix and a compression ratio, and the parameter restoration matrix includes a second data transformation matrix and a restoration ratio. The parameter simplification matrix is ​​determined based on the level classification rules and the fluctuation range of the participant data. The fluctuation range of the participant data is determined based on the upper and lower limits of the data. The participant model and parameter simplification matrix are distributed to the first participant and the second participant. The first participant and the second participant train the participant model based on the second training data and obtain local parameters. After simplifying the local parameters according to the parameter simplification matrix, the simplified parameters are formed and uploaded. The system receives simplified parameters uploaded by the first and second participants, and restores the simplified parameters according to the parameter restoration matrix for comprehensive analysis to obtain global parameters. Based on the global parameters, the parameter update information for the first and second participants is determined and sent to the first and second participants for the next round of iterative training until a trained global model is obtained.

10. The system according to claim 9, characterized in that: The process of restoring simplified parameters based on the parameter restoration matrix includes: The simplified parameters of the first and second participants are combined, and the simplified parameters are restored based on the parameter restoration matrix. When the restoration of the first simplified parameter of the target participant fails, a parameter acquisition instruction is sent to the target participant so that the target participant generates a second simplified parameter based on the parameter acquisition instruction and local parameters, and uploads it. The data volume of the second simplified parameter is greater than the data volume of the first simplified parameter. Receive the second simplified parameters and restore the parameters. When the parameter restoration is successful, update the parameter simplification matrix and the parameter restoration matrix, and send the updated parameter simplification matrix to the target participant to update the target participant's parameter simplification matrix so that simplified parameters can be generated based on the updated parameter simplification matrix in the next iteration.

Images