Training methods, devices, and storage media for image classification models

By combining federated learning and encryption technology, and using data from multiple hospitals to jointly train a dermatology diagnostic model, the problems of existing systems relying on local data and privacy leaks are solved, achieving efficient and accurate dermatology diagnosis and privacy protection.

CN115705687BActive Publication Date: 2026-11-17CHINA MOBILE CHENGDU INFORMATION & TELECOMM TECH CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202110910198.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-09
Publication Date
2026-11-17
Estimated Expiration
2041-08-09

AI Technical Summary

Technical Problem

Existing dermatology diagnostic systems rely on local data, cannot access information from other hospitals, and pose a risk of privacy data leakage when deployed on the Internet. Furthermore, existing deep learning models are influenced by doctors' experience and are time-consuming and labor-intensive.

Method used

By using federated learning and encryption technology, an image classification model is jointly trained using training data from a first device and a second device. By combining plaintext and ciphertext data, adding user information labels, identifying erroneous data, and repeating the training, the model's accuracy and privacy protection are improved.

Benefits of technology

It improves the efficiency and accuracy of dermatology diagnosis, reduces reliance on doctors' experience, enhances data security and privacy protection, and shortens training time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115705687B_ABST
    Figure CN115705687B_ABST
Patent Text Reader

Abstract

This invention provides a training method, apparatus, and storage medium for an image classification model. The method includes: training an image classification model using first training data; wherein the first training data includes at least a first sample image; identifying the first training data using the trained image classification model to obtain a recognition result; determining second training data that the trained image classification model has misidentified based on the recognition result; and inputting at least a portion of the second training data into the image classification model to continue training the image classification model. Thus, during the training process, images that the current image classification model misidentifies are identified, and the training on these inaccurate images is repeated multiple times, thereby accelerating the convergence of the image classification model and improving the training speed of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of image processing and deep learning, and in particular to a training method, apparatus and storage medium for an image classification model. Background Technology

[0002] In clinical practice, besides general visual screening, early diagnosis of skin diseases often relies on dermatological imaging. Current diagnostic methods for skin diseases primarily depend on visual inspection by doctors and computer-aided diagnosis. Compared to other skin imaging techniques, dermoscopy, by eliminating surface reflections, can visualize deeper lesions that are invisible to the naked eye. Compared to visual examination, it can increase diagnostic sensitivity by 10% to 30%, and to some extent reduces the biopsy rate. However, dermoscopic image analysis based on manual identification is not only time-consuming and labor-intensive, but the diagnostic results are also easily influenced by subjective factors such as the doctor's experience.

[0003] Utilizing computer-aided diagnosis (CAD) systems to automatically identify and classify skin diseases based on dermoscopic images can improve diagnostic efficiency and accuracy, enabling precise and detailed classification. This allows doctors to tailor optimal treatment plans based on the specific clinical manifestations of different lesions. However, deploying CAD systems locally in hospitals only allows access to local hospital data, lacking information on patients' visits to other hospitals, resulting in limited model training data and incomplete access to patient history. Furthermore, most current deep learning models and methods for classifying and diagnosing skin diseases require patient lesion images, medical history, and other sensitive information. Deploying CAD systems over the internet could easily lead to the leakage of patient privacy data, resulting in low data privacy and security. Summary of the Invention

[0004] This invention provides a training method, apparatus, and storage medium for an image classification model.

[0005] The technical solution of this invention is implemented as follows:

[0006] This invention provides a method for training an image classification model, the method comprising:

[0007] An image classification model is trained using first training data; wherein the first training data includes at least: a first sample image;

[0008] The recognition result is obtained by using the trained image classification model to identify the first training data;

[0009] Based on the recognition results, the second training data that the trained image classification model has incorrectly identified is determined;

[0010] At least a portion of the second training data is input into the image classification model to continue training the image classification model.

[0011] In the above scheme, determining the second training data for errors in the image classification model based on the recognition result includes:

[0012] Based on the image classification model obtained through training, determine the loss value of the recognition result corresponding to the first sample image;

[0013] The first sample image whose identification result is incorrect and whose loss value is greater than the first preset value is determined as the second training data.

[0014] In the above scheme, the first training data also includes: user information corresponding to the first sample image.

[0015] In the above scheme, training the image classification model using the first training data includes:

[0016] Generate user tags based on the user information;

[0017] The target region and user label are extracted from the first sample image, and the target region and user label are input into the image classification model.

[0018] In the above scheme, the method is executed by the first device, and the first training data includes: a first type of training data from the first device, and / or, a second type of training data from the second device;

[0019] The step of training the image classification model using the first training data includes:

[0020] The image classification model is trained using plaintext training data of the first class.

[0021] And / or,

[0022] The image classification model is trained using the second type of encrypted training data; wherein the second device is any device different from the first device.

[0023] The method in the above scheme further includes:

[0024] Receive encryption key from a third device, wherein the third device is an intermediate device connected to both the first device and the second device;

[0025] Request the second type of training data from the third device;

[0026] The method further includes:

[0027] After inputting the second type of training data into the image classification model, the predicted label output by the image classification model is obtained;

[0028] The prediction tag is encrypted using the encryption key, and the encrypted prediction tag is sent to the third device. The encrypted prediction tag is used by the third device to forward the prediction tag to the second device and to determine the loss value for the first device.

[0029] Receive the loss value provided by the second device returned by the third device;

[0030] Based on the loss value, determine whether to continue training the image classification model.

[0031] In the above scheme, the step of inputting at least a portion of the second training data into the image classification model to continue training the image classification model includes:

[0032] If the training times of a batch of the first sample data have not reached the predetermined number of times and the second training data exists, the image classification model will continue to be trained using the second training data.

[0033] The method further includes:

[0034] When the training times of a batch of the first sample data reach the predetermined number, the training of the image classification model using this batch of the first sample data is terminated.

[0035] This invention also provides a training device for an image classification model, the device comprising: a first training module, a recognition module, a first determination module, and a second training module;

[0036] The first training module is used to train an image classification model using first training data; wherein the first training data includes at least: a first sample image;

[0037] The recognition module is used to identify the first training data through the trained image classification model to obtain a recognition result;

[0038] The first determining module is used to determine, based on the recognition result, the second training data in which the trained image classification model has made a recognition error;

[0039] The second training module is used to input at least a portion of the second training data into the image classification model and continue training the image classification model.

[0040] In the above scheme, the first determining module is specifically used to: determine the loss value of the recognition result corresponding to the first sample image based on the trained image classification model;

[0041] The first sample image whose identification result is incorrect and whose loss value is greater than the first preset value is determined as the second training data.

[0042] In the above scheme, the first training module is specifically used to: determine the user information corresponding to the first sample image.

[0043] In the above scheme, the first training module is specifically used to: generate user labels based on the user information; extract the target region and user labels from the first sample image, and input the target region and user labels into the image classification model.

[0044] In the above scheme, the device is applied to the first device, and the first determining module is specifically used to: determine the first type of training data originating from the first device, and / or the second type of training data originating from the second device;

[0045] The image classification model is trained using plaintext first-class training data; and / or, the image classification model is trained using ciphertext second-class training data; wherein the second device is any device different from the first device.

[0046] In the above scheme, the device further includes: a receiving module, a request module, a prediction module, an encryption module, and a judgment module;

[0047] The receiving module is used to receive an encryption key from a third device, wherein the third device is an intermediate device connected to the first device and the second device respectively;

[0048] The request module is used to request the second type of training data from the third device;

[0049] The prediction module is used to input the second type of training data into the image classification model and obtain the predicted label output by the image classification model.

[0050] The encryption module is used to encrypt the prediction tag using the encryption key and send the encrypted prediction tag to the third device, wherein the encrypted prediction tag is used by the third device to forward it to the second device and determine the loss value for the first device;

[0051] The receiving module is used to receive the loss value provided by the second device returned by the third device;

[0052] The judgment module is used to determine whether to continue training the image classification model based on the loss value.

[0053] In the above scheme, the second training module is specifically used for:

[0054] If the training times of a batch of the first sample data have not reached the predetermined number of times and the second training data exists, the image classification model will continue to be trained using the second training data.

[0055] The second training module is further configured to: terminate the training of the image classification model using the first batch of sample data when the number of training iterations of a batch of the first sample data reaches the predetermined number of iterations.

[0056] This invention also provides a training apparatus for an image classification model, the apparatus comprising: a processor and a memory for storing a computer program capable of running on the processor;

[0057] The processor is used to execute the steps of any of the above-described image classification model training methods when running the computer program.

[0058] This invention also provides a computer storage medium, characterized in that the computer storage medium stores computer-executable instructions; after being executed by a processor, the computer-executable instructions can implement the steps of the above-described image classification model training method.

[0059] In this embodiment, after training the image classification model using the first training data, the trained image classification model is used to identify the first training data, determine the second training data that the currently trained image classification model has misidentified, and input at least a portion of the second training data into the image classification model to continue training the image classification model. In this way, during the training process, images that the current image classification model misidentifies are identified, and the training images that the current image classification model misidentifies are repeatedly trained multiple times, thereby accelerating the convergence of the image classification model and improving the training speed of the model. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0061] Figure 1 A flowchart illustrating the training method for the image classification model provided by this invention;

[0062] Figure 2 A schematic flowchart illustrating the training method for the image classification model provided in an embodiment of the present invention;

[0063] Figure 3This is a schematic diagram of the data interaction process during the construction of the image classification model network provided in an embodiment of the present invention;

[0064] Figure 4 A flowchart illustrating the method for determining the second type of training data provided in an embodiment of the present invention;

[0065] Figure 5 A schematic diagram of the structure of a training device for an image classification model provided in an embodiment of the present invention;

[0066] Figure 6 A schematic diagram of the structure of a training device for another image classification model provided in an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Unless otherwise specified, the embodiments and features described herein can be combined arbitrarily. The steps illustrated in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0070] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms used in communication involved in the embodiments of the present invention will be explained.

[0071] This invention provides a method for training an image classification model. Figure 1 This is a flowchart illustrating the training method for the image classification model provided by the present invention; as shown. Figure 1 As shown, the method includes:

[0072] Step S101: Train an image classification model using the first training data; wherein the first training data includes at least: a first sample image;

[0073] In this embodiment, the first training data is image images, which may be patient image images stored in a local database. For example, the first sample image may be an image image of the patient's lesion tissue, and the image image type includes, but is not limited to, computed tomography (CT) images, digital subtraction angiography (DSA) images, and nuclear magnetic resonance imaging (NMRI).

[0074] Furthermore, the first training data also includes: user information corresponding to the first sample image.

[0075] Specifically, the user information corresponding to the first sample image may include, but is not limited to, at least one of the following:

[0076] Patient age corresponding to the imaging images;

[0077] Patient gender;

[0078] Medical history information;

[0079] Disease information.

[0080] Furthermore, training the image classification model using the first training data includes: generating user labels based on the user information; extracting the target region and user labels from the first sample image; and inputting the target region and user labels into the image classification model.

[0081] Specifically, the user tag is used to indicate the user's basic attributes; the target area can be the lesion area in the image of the patient's lesion tissue, and the target area is the area where the lesion tissue is located.

[0082] Specifically, the region where the lesion is located, labeled with the lesion tag, is extracted from the first sample image using an image segmentation method. The extracted region where the lesion is located, labeled with the lesion tag, and the user tag are input into the image classification model to train the image classification model.

[0083] In this embodiment, the user label information corresponding to the sample image and the target region are used as input data for training the image classification model. Unlike the prior art, which only uses the image as input to the image classification model, this embodiment adds patient age, gender, medical history information, etc. to the classification network of the image classification model, introduces multi-dimensional training data to train the model, and improves the accuracy of classification.

[0084] Step S102: Obtain the recognition result by identifying the first training data through the trained image classification model;

[0085] Specifically, the image classification model obtained through training is the image classification model obtained after training the image classification model based on a batch of the first training data; the first training data is the first training data of the current batch; and the recognition result is the predicted label output by the trained image classification model after recognizing the first training data of the current batch.

[0086] Specifically, after the image classification model is trained based on a batch of the first training data, the first training data of the current batch is input into the trained image classification model for recognition, and the predicted label corresponding to the first training data of the current batch is obtained.

[0087] Step S103: Based on the recognition results, determine the second training data in which the trained image classification model has made incorrect recognitions;

[0088] Specifically, the predicted label corresponding to the first training data in the current batch is compared with the lesion label labeled in the first training data to determine whether the predicted label is correct; if it is incorrect, the incorrect first training data is used as the second training data; the lesion label here can be the actual label of the training data, it can be a label manually labeled by doctors or others, or it can be a label labeled by other models but deemed correct.

[0089] Further, determining the second training data for which the trained image classification model has identified errors based on the recognition results includes: determining the loss value of the recognition result corresponding to the first sample image based on the trained image classification model; and determining the first sample image with an incorrect recognition result and a loss value greater than a first preset value as the second training data.

[0090] Specifically, the first preset value can be a loss value defined by the image classification model or a user-defined loss value; no specific limitation is made here.

[0091] It should be noted that the first preset value is a positive value that is not zero; in some embodiments, the first preset value may be: sorting the loss values ​​of the first training data in the current batch, and taking the Nth loss value of the first training data in the current batch that ranks first as the first preset value, where N is a positive integer greater than zero.

[0092] In other embodiments, the first preset value may be: the maximum value among the loss values ​​of the first training data in the current batch, multiplied by a preset ratio to obtain the preset value.

[0093] Specifically, the difference between the predicted label of the first sample image and the lesion label annotated in the first sample image is determined using the loss function in the trained image classification model, thereby determining the loss value corresponding to the first sample image. The greater the difference between the label identified by the image classification model and the original label information of the first sample image, the greater the loss value.

[0094] In some embodiments, when it is determined that the predicted label corresponding to the first training data in the current batch is incorrect, the first training data corresponding to the incorrect predicted label is used as the second training data; in other embodiments, the loss values ​​of the first training data in the current batch are sorted, and the loss value of the first training data in the current batch that ranks first is used as a first preset value, where N is a positive integer greater than zero; the first training data in the current batch whose loss value is greater than the first preset value is determined; and it is determined whether the predicted label of the first training data in the current batch whose loss value is greater than the first preset value is correct, and the first training data in the current batch whose loss value is greater than the first preset value and whose predicted label is incorrect is used as the second training data.

[0095] In other embodiments, the maximum value of the loss value of the first training data in the current batch is determined, and the maximum value of the loss value of the first training data in the current batch is calculated according to a preset ratio to obtain a first preset value; the first training data in the current batch with a loss value greater than the first preset value is determined, and it is determined whether the predicted label of the first training data in the current batch with a loss value greater than the first preset value is correct; the first training data in the current batch with a loss value greater than the first preset value and a predicted label that is incorrect is used as the second training data.

[0096] Step S104: Input at least a portion of the second training data into the image classification model and continue training the image classification model.

[0097] Specifically, the at least part of the second training data may be a portion of the second training data selected by the user, or it may be the second training data characterizing the same type of lesion tissue, or it may be all of the second training data.

[0098] In this embodiment, after training the image classification model using the first training data, the trained image classification model is used to identify the first training data, determine the second training data that the trained image classification model has misidentified, and input at least a portion of the second training data into the image classification model to continue training the image classification model. In this way, during the training process, images that the current image classification model misidentifies are identified, and the training images that the current image classification model misidentifies are repeatedly trained, thereby accelerating the convergence of the image classification model and improving the training speed of the model.

[0099] Further, in step S101, the first training data includes: a first type of training data from the first device, and / or, a second type of training data from the second device; training the image classification model using the first training data includes: training the image classification model using the plaintext first type of training data; and / or, training the image classification model using the ciphertext second type of training data; wherein, the second device is any device different from the first device.

[0100] Specifically, when the first training data is training data from the first device, the first type of training data includes, but is not limited to, patient image images stored in the local database of the first device; the plaintext first type of training data is the user information, user tags, and target region corresponding to the first type of training data in plaintext state, and the image classification model is trained using the plaintext first type of training data.

[0101] When the data from the first device is insufficient, data from a second device can be introduced, thus increasing the number of samples for the image classification model. Image classification models trained in this way have strong generalization ability and accurate classification results.

[0102] In this embodiment of the disclosure, the second type of training data is encrypted training data. The encrypted training data is ciphertext, but it can still reflect the characteristics of lesions, etc. This can ensure the security of the data on the one hand, and increase the number of training samples for the image classification model on the other hand.

[0103] For example, using training images as lesion blocking samples, encryption can be performed only on the image content that can identify the patient, while the imaging area of ​​the lesion tissue is not encrypted. Alternatively, the position of the lesion tissue in the image can be adjusted to generate new training images.

[0104] When the first training data is a second type of training data originating from the second device, the second type of training data consists of encrypted patient image images stored in the local database of the second device. The encrypted second type of training data consists of the user information, user tags, and target region corresponding to the encrypted first type of training data. The image classification model is trained using the encrypted second type of training data. Here, the user information corresponding to the second type of training data is the same as the user information corresponding to the first type of training data, that is, the second type of training data and the first type of training data are patient image images of the same patient in different hospitals. The encryption method can be homomorphic encryption, RSA encryption, key sharing encryption, etc., which are not specifically limited here.

[0105] Furthermore, when the first training data is second type training data originating from the second device, the method further includes: receiving an encryption key from a third device, wherein the third device is an intermediate device connected to both the first device and the second device; and requesting the second type of training data from the third device.

[0106] Specifically, the second type of training data is encrypted third training data; the third training data is the user's training data in the second device from the first training data; here, the third training data includes at least: a second sample image labeled with lesion tags and user information corresponding to the second sample image; the lesion tags in the second type of training data are lesion tags represented in mask form; the mask data is new data represented in binary obtained by bitwise or logical operations on the original lesion tags.

[0107] In this embodiment, by representing the lesion labels marked in the second sample image with mask data, the security of the training data during the interaction process can be improved, and the privacy information of patients can be protected to a great extent.

[0108] Here, the third device is a federated learning model, which can effectively help multiple organizations to conduct joint modeling without sharing data while meeting the requirements of user privacy protection, data security and government regulations. Parameters can be exchanged through the encryption mechanism in the federated system.

[0109] Specifically, requesting the second type of training data from the third device means:

[0110] The first device requests an encryption key from the third device, uses the encryption key to hash and encrypt the first type of training data, generates a first encrypted dataset, and sends it to the third device. The second device also requests the encryption key from the third device, uses the encryption key to hash and encrypt the sample data in the second device, generates a second encrypted dataset, and sends it to the third device. The sample data in the second device includes at least: sample images in the second device and user information corresponding to the sample images in the second device.

[0111] The third device performs an intersection analysis on the first encrypted data set and the second encrypted data set to determine the data that is identical in both sets. Based on this identical data, an intersection set is determined. After the third device sends the intersection set to the second device, the second device performs a sequential comparison of the sample data in the second device based on the intersection set to determine the user's training data in the second device from the first training data, which is the third training data. The second device then re-encrypts the third training data using the encryption key to generate the second type of training data.

[0112] Specifically, in some embodiments, the same data can be the same encrypted first sample image and encrypted second sample image; the second encrypted data corresponding to the sample image in the second device that is the same as the first sample image is determined as the data in the intersection set;

[0113] In other embodiments, the same data may also be the same user information corresponding to the encrypted first sample image and the same user information corresponding to the encrypted second sample image; the second encrypted data corresponding to the user information of the sample image in the second device that is the same as the user information corresponding to the first sample image is determined as the data in the intersection set.

[0114] Since the first device and the second device use the same encryption key, the plaintext data corresponding to the determined intersection set is also the same. Therefore, the image images of the same patient in the first device and the second device can be identified.

[0115] Here, the second encryption set is generated by hashing the sample images in the second device and the user information corresponding to the sample images in the second device in sequence. In this embodiment, by hashing the first device and the second device, based on the irreversible and decryptable characteristics of hash encryption, the hospital data is encrypted before interaction, and the encryption method that cannot be decrypted is used to ensure data privacy and security.

[0116] The following describes the training method of the image classification model provided in this embodiment of the invention using a specific example. Figure 2 This is a schematic flowchart of the training method for the image classification model provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the data interaction process during the construction of the image classification model network provided in an embodiment of the present invention. Figure 4 A flowchart illustrating the method for determining the second type of training data provided in an embodiment of the present invention;

[0117] The data interaction during the construction of the image classification model network in this embodiment of the invention will be described using a specific example, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the data interaction process during the construction of the image classification model network provided in an embodiment of the present invention;

[0118] A backbone network is established for the image classification model. The backbone of the image classification model comprises at least one convolutional layer, at least one global max pooling layer, at least one fully connected layer, batch normalization, and corrected linear units, etc., and is used for feature extraction from skin disease images. The N-dimensional vector output by the backbone is concatenated with information such as age and gender to form an N+2-dimensional vector. This new vector is then input into a fully connected layer (FC), and finally, the loss value for the image is obtained by passing it through a sigmoid function.

[0119] The Backbone network can be divided into three specific parts, the first part of which is:

[0120] The input is a 224*224 three-channel RGB image. First, a convolutional kernel of size 3*3 with a stride of 1 is passed in. There are 64 convolutional kernels in this layer for sampling. After sampling through two consecutive identical convolutional layers, the output is a 224*224*64 feature map.

[0121] After passing through the previous convolutional layer, it is passed into the next max pooling layer for a 2*2 pooling operation, and the output is a feature map of 112*112*64.

[0122] Similarly, the subsequent convolutional and pooling layers are similar to the previous step, except that the conv3-64 convolutional kernel is replaced with a conv3-128 convolutional kernel. Accordingly, the feature map output by the convolutional layer is 112*112*128, and the feature map output by the pooling layer is 56*56*128.

[0123] Furthermore, the input convolutional kernel size is 3*3 with a stride of 1; there are 256 convolutional kernels sampling in this layer. This stage consists of 3 consecutive layers. After sampling through two identical convolutional layers, a 56*56*256 feature map is obtained. The output of the previous convolutional layer is then fed into a max pooling layer to obtain a 28*28*256 feature map.

[0124] Similarly, the subsequent convolutional and pooling layers operate similarly to the above, except that the conv3-256 convolutional kernel is replaced with a conv3-512 convolutional kernel. Accordingly, the feature map output by the convolutional layer is 56*56*512, and the feature map output by the pooling layer is 28*28*512.

[0125] Finally, the 28*28*512 feature map is fed into a conv3-1024 convolutional kernel with a stride of 2, resulting in a feature map of size 28*28*1024. This concludes the first part of the network.

[0126] The second part of the network:

[0127] The input to the second part of the network is fed into a convolutional layer consisting of conv3-1024 kernels with a stride of 1, resulting in a 28*28*1024 feature map. Figure 4 The output 1 in the network is passed to the next layer as part of the fully connected layer, and then processed in the third part of the network.

[0128] The results from the previous layer are continuously convolved into the next layer to obtain further features. After passing through conv3-512 convolutional layers with strides of 1 and 2, a 28*28*512 feature map is obtained, which is... Figure 4 Output 2. The 28*28*512 feature map is then fed into subsequent convolutional layers as part of the third part of the network for processing.

[0129] Furthermore, after passing through conv3-256 convolutional layers with strides of 1 and 2, a 28*28*256 feature map is obtained, as shown in output 3 of the above figure. This feature map is then fed into subsequent convolutional layers, serving as part of the third part of the network for processing.

[0130] Finally, after convolution with strides of 1 and 2 (conv3-128), the resulting two 28*28*128 feature maps are... Figure 4 Outputs 4 and 5 are used. Both 28*28*128 feature maps are used as input values ​​for the third part of the network; at this point, the second part of the network ends.

[0131] The third part of the network:

[0132] The feature map obtained from the second part of the previous network is combined with information such as age and medical history in the patient's case to form a new feature vector parameter. After global average pooling, it is fed into the fully connected layer. After activation by the sigmoid function, a multi-dimensional feature vector can be obtained, which is used to determine the loss value for various disease classifications.

[0133] The following describes the training method of the image classification model provided in this embodiment of the invention using a specific example. Figure 3 This is a flowchart illustrating the training method for the image classification model provided in an embodiment of the present invention, as shown below. Figure 3 As shown:

[0134] Step 1: Establish the network for the image classification model and preprocess the training data.

[0135] Step 2: Train the image classification model using the first type of training data from the first hospital in plaintext.

[0136] Step 3: Input the first type of training data of the current batch into the image classification model for recognition, and identify the training images with low confidence in the recognition process of the trained image classification model.

[0137] Specifically, the loss value and the recognition result corresponding to each training image are determined. The training images with low confidence are formed into a separate image sequence and then added back into the input image. Additional training is performed on this part of the images to enhance the model's learning of images with low confidence.

[0138] Here, the criteria for judging images with low confidence levels are:

[0139] a) During the training of the image classification model, after each epoch, the training image is imported into the current image classification model for prediction. If the prediction is incorrect, the training image is considered to meet the first condition of low confidence.

[0140] b) During the training of the image classification model, each training image in each epoch has a corresponding loss value. The top 20 images with higher loss values ​​are selected and their intersection is taken with the training images that meet the first condition of low confidence. The images in the intersection can be considered as having low confidence.

[0141] Step 4: Input the training images with low confidence into the current image classification model and continue to execute Step 2.

[0142] Step 5: After each epoch, train the image classification model using the second type of training data from the second hospital in the encrypted file.

[0143] Specifically, such as Figure 4 As shown, Figure 4 A flowchart illustrating the method for determining the second type of training data provided in an embodiment of the present invention;

[0144] The first type of training data in the first hospital is denoted as ID1, where ID1 = {u1, u2, u3...u...} A The training data in the second hospital is represented as ID2, where ID2 = {v1, v2, v3...v...} B};

[0145] Here, u1 includes at least the first sample image and the user information corresponding to the first sample image; v1 includes at least the sample image in the second device and the user information corresponding to the sample image in the second device.

[0146] Hash serialization of ID1 yields dataset A: H(ID1) = {H(u1), H(u2), H(u3)...H(u...}} A H(ID2) = {H(v1), H(v2), H(v3)...H(v4)} is hashed and serialized to obtain dataset B: H(ID2) = {H(v1), H(v2), H(v3)...H(v4)}. B )}

[0147] Each sequence in H(ID1) is encrypted using RSA to obtain Here, rie represents the e-th power of the RSA private key of the i-th element in dataset A; similarly, each sequence in H(ID2) is also encrypted using RSA, resulting in... Since hash encryption is irreversible and cannot be decrypted, it meets the hospital's data transmission requirements.

[0148] The second hospital sends ID'2 to the federated learning model, and the first hospital sends ID'1 to the federated learning model. The federated learning model calculates the intersection of ID'2 and ID'1 to determine the intersection set. Because the first and second hospitals use the same RSA encryption key, calculating the intersection of ID'2 and ID'1 can determine the same sequence in H(ID1) and H(ID2). Here, data is encrypted sequentially. After determining the same sequence in H(ID1) and H(ID2), based on H(ID2), the determined same sequence is compared with data ID2 to determine the medical records of the same patient in the first and second hospitals.

[0149] For example: the user information corresponding to the first sample image of u1 in the first hospital ID1 is the same as the user information corresponding to the sample image of v1 in the second device in the second hospital ID2; the user information corresponding to the first sample image of u3 in the first hospital ID1 is the same as the user information corresponding to the sample image of v3 in the second device in the second hospital ID2; then in the federated learning model, after taking the intersection of ID'2 and ID'1, we will get ID'1∩ID'2={r1 e H(u1)r1 e H(v1),0,r3 e H(u3)r3 e H(v3)...,0}, ID'1∩ID'2 and H(ID2)={H(v1),H(v2),H(v3)...H(v B By comparing them in order, the third training data H = {H(v1), H(v3)} can be determined. The second device re-encrypts the third training data using the encryption key sent by the federated learning model to generate the second type of training data. The federated learning model sends the determined second type of training data to the first hospital, which identifies the second type of training data and updates the weights.

[0150] Specifically, the First Hospital initializes the weights θ A The Second Hospital masked the lesion labels marked by doctors in v1 and generated... The lesion labels marked by doctors in v3 are masked to generate... The second type of training data H' = {r1} is generated by re-encrypting H = {H(v1), H(v3)} using the encryption key. e H(v1),r3 e H(v3)}, where, Generated after encryption Generated after encryption The First Hospital used an image classification model to analyze r1 in the second type of training data. e H(v1) is used for identification to generate predicted labels. r3 in the second type of training data e H(v3) is used for identification to generate predicted labels. Through encryption key pair Generated after encryption right Generate after encryption The First Hospital will and Send it to the Second Hospital, and the Second Hospital will then... and Calculate the loss value Will and For comparison, intermediate variables The Second Hospital and Calculate the loss value Will and By making comparisons, the intermediate variables were calculated. Loss values ​​corresponding to all the second type of training data and intermediate variables The data is sent to the federated learning model, which decrypts the loss value and intermediate variables to generate... and The First Hospital and Calculate the gradient add

[0151] Step 6: Determine the number of training iterations for the image classification model;

[0152] If the image classification model has not reached the preset number of training iterations and there are training images with low confidence, proceed to step 2. If the preset number of training iterations has been reached, proceed to step 7.

[0153] Step 7: End training.

[0154] like Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a training device for an image classification model provided in an embodiment of the present invention. The device includes: a first training module 501, a recognition module 502, a first determination module 503, and a second training module 504.

[0155] The first training module 501 is used to train an image classification model using first training data; wherein, the first training data includes at least: a first sample image;

[0156] The recognition module 502 is used to recognize the first training data by the trained image classification model to obtain a recognition result.

[0157] The first determining module 503 is used to determine, based on the recognition result, the second training data in which the trained image classification model has made a recognition error;

[0158] The second training module 504 is used to input at least a portion of the second training data into the image classification model to continue training the image classification model.

[0159] Specifically, the first determining module 503 is further configured to: determine the loss value of the recognition result corresponding to the first sample image based on the image classification model obtained by training; and determine the first sample image with an incorrect recognition result and a loss value greater than a first preset value as the second training data.

[0160] Specifically, the first training module 501 is further configured to: determine the user information corresponding to the first sample image.

[0161] Specifically, the first training module 501 is further configured to: generate user labels based on the user information; extract the target region and user labels from the first sample image, and input the target region and user labels into the image classification model.

[0162] Specifically, the device is applied to a first device, and the first determining module 503 is further configured to: determine a first type of training data originating from the first device, and / or a second type of training data originating from a second device; train the image classification model using the plaintext first type of training data; and / or train the image classification model using the ciphertext second type of training data; wherein the second device is any device different from the first device.

[0163] Specifically, the device further includes: a receiving module 505, a request module 506, a prediction module 507, an encryption module 508, and a judgment module 509;

[0164] The receiving module 505 is used to receive an encryption key from a third device, wherein the third device is an intermediate device connected to the first device and the second device respectively;

[0165] The request module 506 is used to request the second type of training data from the third device;

[0166] The prediction module 507 is used to input the second type of training data into the image classification model and obtain the predicted label output by the image classification model.

[0167] The encryption module 508 is used to encrypt the prediction tag using the encryption key and send the encrypted prediction tag to the third device, wherein the encrypted prediction tag is used by the third device to forward it to the second device and determine the loss value for the first device;

[0168] The receiving module 505 is also used to receive the loss value provided by the second device returned by the third device;

[0169] The judgment module 509 is used to determine whether to continue training the image classification model based on the loss value.

[0170] Specifically, the second training module 504 is further configured to: continue training the image classification model using the second training data when the number of training iterations of a batch of the first sample data has not reached a predetermined number and the second training data exists; the second training module 504 is further configured to: end the training of the image classification model using the current batch of the first sample data when the number of training iterations of a batch of the first sample data reaches the predetermined number of times.

[0171] To implement the method of the embodiments of the present invention, the embodiments of the present invention provide another training device for an image classification model, specifically, as follows: Figure 6 As shown, Figure 6 A schematic diagram of the structure of a training device for another image classification model provided in an embodiment of the present invention; the device 60 includes a processor 601 and a memory 602 for storing computer programs that can run on the processor;

[0172] When the processor 601 runs the computer program, it performs the following: training an image classification model using first training data; wherein the first training data includes at least a first sample image; identifying the first training data using the trained image classification model to obtain a recognition result; determining, based on the recognition result, second training data that the trained image classification model has incorrectly identified; and inputting at least a portion of the second training data into the image classification model to continue training the image classification model.

[0173] In one embodiment, the processor 601 is further configured to, when running the computer program, perform the following: determine the loss value of the recognition result corresponding to the first sample image based on the trained image classification model; and determine the first sample image whose recognition result is incorrect and whose loss value is greater than a first preset value as the second training data.

[0174] In one embodiment, the processor 601 is further configured to, when running the computer program, perform: determining user information corresponding to the first sample image that the first training data also includes.

[0175] In one embodiment, the processor 601 is further configured to, when running the computer program, perform: generating user tags based on the user information; extracting a target region and user tags from the first sample image; and inputting the target region and user tags into the image classification model.

[0176] In one embodiment, the processor 601 is further configured to, when running the computer program, execute: the first training data includes: a first type of training data originating from the first device, and / or, a second type of training data originating from a second device; the step of training an image classification model using the first training data includes: training the image classification model using the plaintext first type of training data; and / or, training the image classification model using the ciphertext second type of training data; wherein, the second device is any device different from the first device.

[0177] In one embodiment, the processor 601 is further configured to, when running the computer program, perform the following: receiving an encryption key from a third device, wherein the third device is an intermediate device connected to both the first device and the second device; requesting the second type of training data from the third device; the processor 601 is further configured to, when running the computer program, perform the following: inputting the second type of training data into the image classification model to obtain the predicted label output by the image classification model; encrypting the predicted label using the encryption key and sending the encrypted predicted label to the third device, wherein the encrypted predicted label is used by the third device to forward it to the second device and determine a loss value for the first device; receiving the loss value returned by the third device from the second device; and determining whether to continue training the image classification model based on the loss value.

[0178] In one embodiment, the processor 601 is further configured to, when running the computer program, execute: when the training times of a batch of the first sample data have not reached a predetermined number and the second training data exists, continue to train the image classification model using the second training data;

[0179] The processor 601 is also configured to, when running the computer program, execute: when the number of training iterations of a batch of the first sample data reaches the predetermined number, terminate the training of the image classification model using the current batch of the first sample data.

[0180] It should be noted that the training device for the image classification model provided in the above embodiments and the training method for the image classification model belong to the same concept. For details of its implementation process, please refer to the method embodiments, which will not be repeated here.

[0181] Of course, in practical applications, such as Figure 6As shown, the device 60 may further include at least one network interface 603. The various components in the image classification model training device 60 are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 All buses are labeled as bus system 604. The number of processors 601 can be at least one. Network interface 603 is used for wired or wireless communication between the image classification model training device 60 and other devices.

[0182] In this embodiment of the invention, the memory 602 is used to store various types of data to support the operation of the training device 60 for the image classification model.

[0183] The methods disclosed in the above embodiments of the present invention can be applied to processor 601, or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 602. Processor 601 reads the information in memory 602 and combines its hardware to complete the steps of the aforementioned method.

[0184] In an exemplary embodiment, the training device 60 for the image classification model may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0185] In an exemplary embodiment, the present invention also provides a computer-readable storage medium, such as a memory 602 including a computer program, which can be executed by a processor 601 of an image classification model training device 60 to complete the steps described in the foregoing method.

[0186] Specifically, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it performs the following: training an image classification model using first training data; wherein the first training data includes at least a first sample image; identifying the first training data using the trained image classification model to obtain a recognition result; determining, based on the recognition result, second training data that the trained image classification model has incorrectly identified; and inputting at least a portion of the second training data into the image classification model to continue training the image classification model.

[0187] In one embodiment, when the computer program is run by the processor, it performs the following: determining the loss value of the recognition result corresponding to the first sample image based on the trained image classification model; and determining the first sample image with an incorrect recognition result and a loss value greater than a first preset value as the second training data.

[0188] In one embodiment, when the computer program is run by the processor, it executes the following: the first training data further includes user information corresponding to the first sample image.

[0189] In one embodiment, when the computer program is run by the processor, it performs the following: generating user tags based on the user information; extracting a target region and user tags from the first sample image; and inputting the target region and user tags into the image classification model.

[0190] In one embodiment, when the computer program is run by the processor, it executes the following: the first training data includes: a first type of training data originating from the first device, and / or, a second type of training data originating from a second device; training the image classification model using the first training data includes: training the image classification model using the first type of training data in plaintext; and / or, training the image classification model using the second type of training data in ciphertext; wherein the second device is any device different from the first device.

[0191] In one embodiment, when the computer program is run by the processor, it performs the following actions: receiving an encryption key from a third device, wherein the third device is an intermediate device connected to both the first device and the second device; requesting the second type of training data from the third device; when the computer program is run by the processor, it performs the following actions: inputting the second type of training data into the image classification model to obtain the predicted label output by the image classification model; encrypting the predicted label using the encryption key and sending the encrypted predicted label to the third device, wherein the encrypted predicted label is used by the third device to forward it to the second device and determine a loss value for the first device; receiving the loss value provided by the second device returned by the third device; and determining whether to continue training the image classification model based on the loss value.

[0192] In one embodiment, when the computer program is run by the processor, it executes the following: if the training times of a batch of the first sample data have not reached a predetermined number and the second training data exists, it continues to train the image classification model using the second training data; when the computer program is run by the processor, it executes the following: if the training times of a batch of the first sample data have reached the predetermined number, it terminates the training of the image classification model using the current batch of the first sample data.

[0193] It should be noted that the computer-readable storage medium provided in the embodiments of the present invention may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device including one or any combination of the above-mentioned memories.

[0194] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of the present invention are included within the scope of protection of the present invention.

Claims

1. A training method for an image classification model, characterized in that, The method is performed by a first device, and the method includes: An image classification model is trained using first training data; wherein the first training data consists of patient image pictures, including at least: a first sample image and user information corresponding to the first sample image; the user information is used to generate user labels; the first training data includes: a first type of training data originating from the plaintext usage of the first device and a second type of training data originating from the ciphertext usage of the second device; the user information corresponding to the second type of training data is the same as the user information corresponding to the first type of training data; the second type of training data and the first type of training data are patient image pictures of the same patient in different hospitals; the second device is any device different from the first device. The method further includes, prior to training the image classification model using the first training data, the following steps: receiving an encryption key from a third device, wherein the third device is an intermediate device connected to both the first device and the second device; requesting the second type of training data from the third device, including: encrypting the first type of training data using the encryption key to generate a first encrypted dataset and sending it to the third device; the first encrypted dataset is used by the third device to perform an intersection analysis on the first and second encrypted datasets to determine the data that is identical in both datasets, to determine an intersection set based on the determined identical data, and to send the intersection set to the second device; the second encrypted dataset is sent by the second device to the third device; the intersection set is used by the second device to perform a sequential comparison of sample data in the second device based on the intersection set to determine the user's training data in the second device from the first training data; the second type of training data is generated by the second device re-encrypting the training data in the second device using the encryption key; the second encrypted dataset is generated by the second device requesting the encryption key from the third device and encrypting the sample data in the second device using the encryption key. The recognition result is obtained by using the trained image classification model to identify the first training data; Based on the recognition results, the second training data that the trained image classification model has incorrectly identified is determined; At least a portion of the second training data is input into the image classification model to continue training the image classification model.

2. The training method for the image classification model according to claim 1, characterized in that, The step of determining the second training data for errors in the image classification model based on the recognition result includes: Based on the image classification model obtained through training, determine the loss value of the recognition result corresponding to the first sample image; The first sample image whose identification result is incorrect and whose loss value is greater than the first preset value is determined as the second training data.

3. The training method for the image classification model according to claim 1, characterized in that, The step of training the image classification model using the first training data includes: Generate user tags based on the user information; The target region and user label are extracted from the first sample image, and the target region and user label are input into the image classification model.

4. The training method for the image classification model according to claim 1 or 2, characterized in that, The step of training the image classification model using the first training data includes: The image classification model is trained using plaintext training data of the first class. And / or, The image classification model is trained using the second type of training data in encrypted form.

5. The training method for the image classification model according to claim 4, characterized in that, The method further includes: After inputting the second type of training data into the image classification model, the predicted label output by the image classification model is obtained; the predicted label is encrypted using the encryption key, and the encrypted predicted label is sent to the third device, wherein the encrypted predicted label is used by the third device to forward to the second device and determine the loss value for the first device; the loss value provided by the second device is received by the third device; based on the loss value, it is determined whether to continue training the image classification model.

6. The training method for the image classification model according to claim 1, characterized in that, The step of inputting at least a portion of the second training data into the image classification model to continue training the image classification model includes: If the training times of a batch of first sample data have not reached the predetermined number of times and the second training data exists, the image classification model will continue to be trained using the second training data. The method further includes: When the training times of a batch of the first sample data reach the predetermined number, the training of the image classification model using this batch of the first sample data is terminated.

7. A training device for an image classification model, characterized in that, The device includes: a first training module, an identification module, a first determination module, and a second training module; The first training module is used to train an image classification model using first training data; wherein the first training data is patient image images, including at least: a first sample image and user information corresponding to the first sample image; the user information is used to generate user labels; the first training data includes: a first type of training data with plaintext usage from a first device and a second type of training data with ciphertext usage from a second device; the user information corresponding to the second type of training data is the same as the user information corresponding to the first type of training data; the second type of training data and the first type of training data are patient image images of the same patient in different hospitals; wherein the second device is any device different from the first device; and before training the image classification model using the first training data, receiving an encryption key from a third device, the third device being an intermediate device connected to the first device and the second device respectively; requesting the second type of training data from the third device, including: using the encryption key to encrypt the first type of training data. The training data is encrypted to generate a first encrypted dataset, which is then sent to the third device. The first encrypted dataset is used by the third device to find the intersection of the first and second encrypted datasets, determine the data that is identical in both datasets, determine the intersection set based on the identified identical data, and send the intersection set to the second device. The second encrypted dataset is sent by the second device to the third device. The intersection set is used by the second device to perform a sequential comparison of the sample data in the second device based on the intersection set, determining the user's training data in the second device from the first training data. The second type of training data is generated by the second device re-encrypting the training data in the second device using the encryption key. The second encrypted dataset is generated by the second device requesting the encryption key from the third device and encrypting the sample data in the second device using the encryption key. The recognition module is used to identify the first training data through the trained image classification model to obtain a recognition result; The first determining module is used to determine, based on the recognition result, the second training data in which the trained image classification model has made a recognition error; The second training module is used to input at least a portion of the second training data into the image classification model and continue training the image classification model.

8. The training apparatus for the image classification model according to claim 7, characterized in that, The first determining module is specifically used to: determine the loss value of the recognition result corresponding to the first sample image based on the trained image classification model; The first sample image whose identification result is incorrect and whose loss value is greater than the first preset value is determined as the second training data.

9. The training apparatus for the image classification model according to claim 7, characterized in that, The first training module is specifically used for: generating user labels based on the user information; extracting target regions and user labels from the first sample image; and inputting the target regions and user labels into the image classification model.

10. The training apparatus for the image classification model according to claim 7 or 8, wherein the apparatus is applied to the first device, characterized in that, The first determining module is specifically configured to: train the image classification model using plaintext first-type training data; and / or, train the image classification model using ciphertext second-type training data.

11. The training apparatus for the image classification model according to claim 10, characterized in that, The device further includes: a receiving module, a request module, a prediction module, an encryption module, and a judgment module; The receiving module is used to receive an encryption key from a third device, wherein the third device is an intermediate device connected to the first device and the second device respectively; The request module is used to request the second type of training data from the third device; The prediction module is used to input the second type of training data into the image classification model and obtain the predicted label output by the image classification model. The encryption module is used to encrypt the prediction tag using the encryption key and send the encrypted prediction tag to the third device, wherein the encrypted prediction tag is used by the third device to forward it to the second device and determine the loss value for the first device; The receiving module is used to receive the loss value provided by the second device returned by the third device; The judgment module is used to determine whether to continue training the image classification model based on the loss value.

12. The training apparatus for the image classification model according to claim 7, characterized in that, The second training module is specifically used for: If the training times of a batch of first sample data have not reached the predetermined number of times and the second training data exists, the image classification model will continue to be trained using the second training data. The second training module is also used for: When the training times of a batch of the first sample data reach the predetermined number, the training of the image classification model using this batch of the first sample data is terminated.

13. A training device for an image classification model, characterized in that, The apparatus includes: a processor and a memory for storing a computer program capable of running on the processor; wherein, when the processor runs the computer program, it performs the steps of the method according to any one of claims 1 to 6.

14. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Training data processing method, classification model training method and device

    CN110163849A

  • Training method, device and equipment and storage medium

    CN112132239A