Classification model training method and device, equipment, storage medium and program product

Abstract

The invention relates to a classification model training method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining first image data without labels and second image data with labels, inputting the first image data and the second image data into a preset classification model for training, and outputting a first classification result of the first image data, a first classification sub-model corresponding to the second image data and third image data, determining a first loss according to the first classification result and a second classification result output by the first classification sub-model, determining a second loss according to the third image data and the first image data, and training a preset classification model according to the second loss and the first loss to obtain a target classification model; the third image data is an image after mask processing and completion processing. According to the method for training the classification model based on the labeled images and the unlabeled images, the problem that the training difficulty of the model is large is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a classification model training method, device, equipment, storage medium and program product. Background Art

[0002] The fundus (including the retina and optic nerve) is the only tissue in the body that can directly and non-invasively observe blood vessels and nerves. Fundus images can not only diagnose eye diseases such as diabetic retinopathy and retinal detachment, but also observe systemic diseases such as hypertension, lupus erythematosus, and Alzheimer's disease.

[0003] With the development of artificial intelligence, training computer vision models with ultra-wide-angle fundus images to enable them to identify various types of diseases has become a hot research topic. However, in the medical field, the labels of the collected ultra-wide-angle fundus image data are usually quite sparse, which increases the difficulty of training computer vision models that rely on labeled data.

[0004] Therefore, how to reduce the difficulty of training computer vision models in eye disease recognition applications has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] Based on this, it is necessary to provide a training method, device, equipment, storage medium and program product for a classification model that can reduce the difficulty of training a computer vision model in order to address the above technical problems.

[0006] In a first aspect, the present application provides a method for training a classification model, comprising:

[0007] Acquire unlabeled first image data and labeled second image data;

[0008] Input the first image data and the second image data into a preset classification model for training, and output a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing;

[0009] Determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and determine a second loss according to the third image data and the first image data;

[0010] According to the second loss and the first loss, the preset classification model is trained to obtain the target classification model.

[0011] In one embodiment, the preset classification model includes a first initial classification sub-model and a second initial classification sub-model, and the first image data and the second image data are input into the preset classification model for training, and the first classification result of the first image data, the first classification sub-model corresponding to the second image data, and the third image data are output, including:

[0012] Inputting the second image data into the first initial classification sub-model for training to obtain a first classification sub-model;

[0013] Inputting the first image data into the first classification sub-model for classification to obtain a second classification result;

[0014] The first image data is input into the second initial classification sub-model for processing to obtain a first classification result and third image data.

[0015] In one embodiment, the second initial classification sub-model includes an encoder and a decoder, and the first image data is input into the second initial classification sub-model for processing to obtain the third image data, including:

[0016] Performing mask processing on the first image data to obtain a masked image;

[0017] Inputting the masked image into an encoder for encoding to obtain an encoded image;

[0018] The encoded image is input into the decoder for completion processing to obtain third image data.

[0019] In one embodiment, the second initial classification sub-model further includes a perceptron, and the first image data is input into the second initial classification sub-model for processing to obtain a first classification result, including:

[0020] The encoded image is input into the perceptron for classification to obtain the first classification result.

[0021] In one embodiment, the masking process is performed on the first image data to obtain the masked image, including:

[0022] Performing block processing on the first image data to obtain a plurality of image blocks;

[0023] Mask processing is performed on some image blocks among the multiple image blocks to obtain a masked image.

[0024] In one embodiment, determining the second loss according to the third image data and the first image data includes:

[0025] Determining a corresponding third image pixel value according to the third image data;

[0026] Determine a corresponding first image pixel value according to the first image data;

[0027] Determine a second loss according to the third image pixel value and the first image pixel value. Step A;

[0028] In a second aspect, the present application also provides a training device for a classification model, comprising:

[0029] An acquisition module, used for acquiring unlabeled first image data and labeled second image data;

[0030] A training module, used for inputting the first image data and the second image data into a preset classification model for training, and outputting a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing;

[0031] a determination module, configured to determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and to determine a second loss according to the third image data and the first image data;

[0032] The training module is used to train the preset classification model according to the second loss and the first loss to obtain a target classification model.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire unlabeled first image data and labeled second image data;

[0035] Input the first image data and the second image data into a preset classification model for training, and output a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing;

[0036] Determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and determine a second loss according to the third image data and the first image data;

[0037] According to the second loss and the first loss, the preset classification model is trained to obtain the target classification model.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0039] Acquire unlabeled first image data and labeled second image data;

[0040] Input the first image data and the second image data into a preset classification model for training, and output a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing;

[0041] Determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and determine a second loss according to the third image data and the first image data;

[0042] According to the second loss and the first loss, the preset classification model is trained to obtain the target classification model.

[0043] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0044] Acquire unlabeled first image data and labeled second image data;

[0045] Input the first image data and the second image data into a preset classification model for training, and output a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing;

[0046] Determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and determine a second loss according to the third image data and the first image data;

[0047] According to the second loss and the first loss, the preset classification model is trained to obtain the target classification model.

[0048] The training method, device, equipment, storage medium and program product of the above classification model first obtain unlabeled first image data and labeled second image data, then input the first image data and the second image data into the preset classification model for training, output the first classification result of the first image data, the first classification sub-model corresponding to the second image data, and the third image data, then determine the first loss according to the first classification result and the second classification result output by the first classification sub-model, and determine the second loss according to the third image data and the first image data, and train the preset classification model according to the second loss and the first loss to obtain the target classification model; the third image data is the image after mask processing and completion processing. The above method provides a method for training a classification model based on labeled images and unlabeled images, which solves the problem that the training of computer vision models is difficult in the current application of eye disease recognition; in addition, the preset classification model is trained based on the first loss determined based on the classification result corresponding to the preset classification model and the second loss determined based on the image information corresponding to the preset classification model, thereby ensuring the accuracy of the classification model obtained after training. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0050] FIG1 is a diagram of an application environment of a classification model training method in one embodiment;

[0051] Figure 2 A schematic diagram of a flow chart of a training method for a classification model in one embodiment;

[0052] Figure 3 A schematic diagram of a flow chart of a training method for a classification model in another embodiment;

[0053] Figure 4 A schematic diagram of a flow chart of a training method for a classification model in another embodiment;

[0054] Figure 5 A schematic diagram of a flow chart of a training method for a classification model in another embodiment;

[0055] Figure 6 A schematic diagram of a flow chart of a training method for a classification model in another embodiment;

[0056] Figure 7 A schematic diagram of a flow chart of a training method for a classification model in another embodiment;

[0057] Figure 8 A schematic diagram of a flow chart of a training method for a classification model in another embodiment;

[0058] FIG9 is a block diagram of a training device for a classification model in one embodiment;

[0059] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The fundus (including the retina and optic nerve) is the only tissue in the body that can be used to directly observe blood vessels and nerves without invasiveness. Fundus images can not only diagnose eye diseases such as diabetic retinopathy and retinal detachment, but also observe systemic diseases such as hypertension, lupus erythematosus, and Alzheimer's disease. Compared with ordinary fundus color photos, ultra-wide-angle fundus images are obtained by sampling the fundus at a wider angle. They not only include the sampling range of traditional fundus color photos, but also can more clearly observe the situation around the fundus, thus facilitating more accurate and comprehensive screening and diagnosis.

[0062] Artificial intelligence has now been widely used in medical image interpretation. Artificial intelligence is an emerging technical science that studies and develops theories, methods, technologies and applications for simulating, extending and expanding human intelligence. Machine learning refers to the process of using certain algorithms to guide computers to use known data to derive appropriate models and use this model to make judgments on new situations. Deep learning is a subfield of machine learning research. Its concept originates from the study of artificial neural networks. Multilayer perceptrons with multiple hidden layers are a type of deep learning structure. Its motivation is to establish an artificial neural network that simulates the human brain for analysis and learning, imitate the mechanism of the human brain to interpret data including images, sounds and text, and combine low-level features to form more abstract high-level features to discover the association and distribution of data.

[0063] With the development of artificial intelligence, training computer vision models with ultra-wide-angle fundus images to enable them to identify various types of diseases has become a hot research topic. However, in the medical field, the labels of the collected ultra-wide-angle fundus image data are usually quite sparse, which increases the difficulty of training computer vision models that rely on labeled data.

[0064] Therefore, how to reduce the difficulty of training computer vision models in eye disease recognition applications has become a technical problem that needs to be solved urgently. This application aims to solve this problem.

[0065] After the above introduction to the background technology of the training method of the classification model provided by the embodiment of the present application, the implementation environment involved in the training method of the classification model provided by the embodiment of the present application will be briefly described below. The training method of the classification model provided by the embodiment of the present application can be applied to the implementation environment as shown in Figure 1, which includes a server 104, and the server 104 can be implemented with an independent server 104 or a server cluster composed of multiple servers 104, and the data storage system 102 can store the data that the server 104 needs to process. The data storage system 102 can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the server 104 can obtain labeled fundus image data and unlabeled fundus image data, and train the classification model based on the labeled fundus image data and the unlabeled fundus image data to obtain a trained classification model, and the trained classification model can determine the disease type corresponding to the fundus image.

[0066] Those skilled in the art will understand that the structure shown in FIG. 1 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0067] After introducing the application scenarios of the training method of the classification model provided in the embodiments of the present application, the following focuses on introducing the training method of the classification model described in the present application.

[0068] In one embodiment, Figure 2 As shown, a training method for a classification model is provided, and the method is applied to the server in FIG1 as an example for explanation, including the following steps:

[0069] S201: Acquire first image data without labels and second image data with labels.

[0070] Among them, the first image data and the second image data are both fundus images, specifically ultra-wide-angle fundus images. The unlabeled first image data refers to the fundus image marked with the disease type, and the labeled second image data refers to the fundus image without the disease type marked.

[0071] In an embodiment of the present application, when it is necessary to train a classification model based on fundus images to identify the disease type corresponding to the fundus images based on the trained classification model, it is necessary to first collect unlabeled first image data and labeled second image data within a historical time period; optionally, the unlabeled first image data can be directly obtained from the fundus image scanning device, and the labeled second image data can be obtained from the doctor's disease diagnosis data.

[0072] S202, inputting the first image data and the second image data into a preset classification model for training, and outputting a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data.

[0073] The preset classification model refers to an initial classification model that has not been trained.

[0074] The third image data is an image after mask processing and completion processing.

[0075] In an embodiment of the present application, after the first image data and the second image data are acquired as described above, the first image data and the second image data can be input into a preset classification model, and the preset classification model can be trained to obtain a first classification result corresponding to the first image data, a first classification sub-model corresponding to the second image data, and third image data.

[0076] Optionally, the preset classification model may include a first initial classification sub-model and a second initial classification sub-model. The first initial classification sub-model can be trained based on the second image data to obtain the first classification sub-model corresponding to the second image data, and the first image data can be input into the second initial classification sub-model for initial classification to obtain a first classification result, and the first image data can be masked and completed in sequence to obtain third image data.

[0077] S203: Determine a first loss according to the first classification result and the second classification result output by the first classification sub-model, and determine a second loss according to the third image data and the first image data.

[0078] In an embodiment of the present application, the first image data can be input into the first classification sub-model for classification to obtain a second classification result, and then the first loss can be determined based on the difference between the first classification result and the second classification result. After the third image data is determined as above, the second loss can be determined based on the difference between the first image data and the third image data.

[0079] S204: Train the preset classification model according to the second loss and the first loss to obtain a target classification model.

[0080] In the embodiment of the present application, after the first loss and the second loss are obtained as described above, the target loss can be determined according to the second loss and the first loss, and the preset classification model can be trained according to the target loss to obtain the target classification model.

[0081] Optionally, the process of determining the target loss according to the first loss and the second loss is as follows:

[0082]

[0083] in, is the target loss, is the first loss, is the loss coefficient, For the second loss.

[0084] It should be noted that the Adam optimizer can be used to perform gradient descent optimization on the model parameters of the preset classification model. When the target loss value is less than the preset loss threshold, it means that the preset classification model has been trained. At this time, the training process can be stopped in time.

[0085] The training method of the classification model provided in the embodiment of the present application first obtains unlabeled first image data and labeled second image data, then inputs the first image data and the second image data into a preset classification model for training, outputs a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and a third image data, then, determines a first loss based on the first classification result and the second classification result output by the first classification sub-model, determines a second loss based on the third image data and the first image data, and trains the preset classification model based on the second loss and the first loss to obtain a target classification model; the third image data is an image after mask processing and completion processing. The above method provides a method for training a classification model based on labeled images and unlabeled images, which solves the problem that the training of computer vision models is difficult in current eye disease recognition applications; in addition, the preset classification model is trained based on the first loss determined based on the classification result corresponding to the preset classification model and the second loss determined based on the image information corresponding to the preset classification model, thereby ensuring the accuracy of the classification model obtained after training.

[0086] In one embodiment, Figure 2 Based on the embodiment shown, the preset classification model includes a first initial classification sub-model and a second initial classification sub-model, and the process of determining the first classification result, the first classification sub-model corresponding to the second image data, and the third image data can be described, such as Figure 3As shown, the above S202 "inputting the first image data and the second image data into a preset classification model for training, and outputting a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and the third image data" includes:

[0087] S301. Input the second image data into the first initial classification sub-model for training to obtain a first classification sub-model.

[0088] The first initial classification sub-model refers to an initial type of the first classification sub-model that can be trained according to the labeled second image data.

[0089] The first initial classification sub-model can be a ResNet-50 model pre-trained based on the ImageNet dataset. The first classification sub-model can classify fundus images. Optionally, the last layer of the neural network of the first classification sub-model can be replaced with an input dimension of 1024 and an output dimension of , so that each dimension of the output corresponds to the judgment probability of the corresponding disease.

[0090] In the embodiment of the present application, after the labeled second image data is acquired as described above, the second image data can be input into the first initial classification sub-model for training to obtain a first classification sub-model after training.

[0091] S302: Input the first image data into the first classification sub-model for classification to obtain a second classification result.

[0092] In the embodiment of the present application, after the trained first classification sub-model is acquired, the first image data can be input into the trained first classification sub-model for classification to obtain a second classification result.

[0093] Optionally, the first image data may be input into the trained first classification sub-model to obtain an output vector , the output vector The value of each dimension represents the probability of the corresponding disease predicted by the first classification sub-model based on the first image data, and sets the disease label with a probability greater than 50% as positive, otherwise it is set to negative.

[0094] S303: Input the first image data into the second initial classification sub-model for processing to obtain a first classification result and third image data.

[0095] The second initial classification sub-model may include a classification sub-unit and an image processing sub-unit.

[0096] In an embodiment of the present application, the first image data can be input into a classification subunit for classification processing to obtain a first classification result, and the first image data can be input into an image processing subunit for image processing to obtain third image data after image processing.

[0097] Optionally, the process of acquiring the first classification result and the process of acquiring the third image data may be described in detail respectively.

[0098] First, the process of obtaining the third image data is described, see Figure 4 The second initial classification sub-model includes an encoder and a decoder, and the above S303 "inputting the first image data into the second initial classification sub-model for processing to obtain third image data" includes:

[0099] S401 . Perform mask processing on first image data to obtain a masked image.

[0100] In the embodiment of the present application, after the first image data is acquired as described above, mask processing may be performed on part of the image data in the first image data to obtain a masked image.

[0101] Optionally, the process of performing mask processing on the first image data may be described, see Figure 5 , that is, the above-mentioned S401 “perform mask processing on the first image data to obtain a masked image” includes:

[0102] S4011. Perform block processing on the first image data to obtain multiple image blocks.

[0103] In the embodiment of the present application, after the first image data is obtained, each image in the first image data may be processed into blocks to obtain a plurality of image blocks corresponding to each image. Optionally, the first image data may be input into the input processing layer to divide each image in the first image data into a plurality of image blocks. For example, according to the length of the image He Kuan , divide the image into image blocks.

[0104] Optionally, before the first image data is processed in blocks, the resolution of the first image data may be adjusted to Then, the adjusted first image data is subjected to data enhancement processing to obtain the first image data after data enhancement processing, and then the first image data after data enhancement processing is subjected to block processing. Optionally, the data enhancement processing method includes standardizing color distribution, randomly cutting to Resolution size, as well as randomly adding horizontal flip, vertical flip and random angle rotation, etc.

[0105] S4012: Perform mask processing on some image blocks among the multiple image blocks to obtain a masked image.

[0106] In the embodiment of the present application, after the multiple image blocks corresponding to an image are obtained, some of the multiple image blocks may be subjected to mask processing to obtain a masked image. For example, 75% of the image blocks of the multiple image blocks corresponding to an image are subjected to mask processing, that is, all pixel values ​​of the masked image blocks are set to 0, thereby obtaining a masked image.

[0107] Thus, the masked image corresponding to the first image data is obtained.

[0108] S402: Input the masked image into an encoder for encoding to obtain an encoded image.

[0109] The encoder may be an encoder composed of a multi-layer Transformer network.

[0110] The encoded image can be of length , which contains high-level semantic information about the ultra-wide-angle fundus image.

[0111] In the embodiment of the present application, after the masked image is obtained, the masked image can be input into an encoder for encoding to obtain an encoded image.

[0112] Optionally, after obtaining the masked image as above, you can first use a size of , the step length is The 2D convolutional model maps the masked image to a latent representation space ,in , and obtain the vector representing the output of the spatial representation. The vector representing the output of the spatial representation is then input into an encoder composed of multiple layers of Transformer for encoding to obtain the encoded image.

[0113] S403: Input the encoded image into a decoder for completion processing to obtain third image data.

[0114] The decoder may be a decoder composed of multiple layers of Transformer.

[0115] In the embodiment of the present application, after the encoded image is acquired as described above, the encoded image may be input into a decoder for completion processing to obtain completed third image data.

[0116] Optionally, in order to apply to the mask image completion task, the above-obtained encoded image needs to be input into the decoder for completion processing to obtain a third image data with the same resolution as the first image data. .

[0117] The above is the process of acquiring the third image data.

[0118] Second, the process of obtaining the first classification result is described, see Figure 6 , the second initial classification sub-model further includes a perceptron, and the above S303 “inputting the first image data into the second initial classification sub-model for processing to obtain a first classification result” includes:

[0119] S404: Input the encoded image into a perceptron for classification to obtain a first classification result.

[0120] In the embodiment of the present application, after the encoded image is obtained as described above, the encoded image can be input into the perception machine. The first classification result is obtained by memory classification.

[0121] The training method of the classification model provided in the embodiment of the present application makes full use of massive unlabeled image data to train the classification model, so that the model after training can have better generalization and robustness in distinguishing various diseases; in addition, the semi-supervised learning method using the first classification sub-model for knowledge distillation can further enhance the efficiency of unlabeled data in the pre-training process, and can further enhance the performance of the base model.

[0122] In one embodiment, Figure 2-Figure 6 Based on any of the embodiments shown, a method for determining the second loss can be described as follows: Figure 7 As shown, the above S203 “determine the second loss according to the third image data and the first image data” includes:

[0123] S501. Determine corresponding third image pixel values ​​according to third image data.

[0124] In the embodiment of the present application, after the third image data is acquired as described above, the pixel value corresponding to the third image data, that is, the third image pixel value, can be determined.

[0125] S502: Determine a corresponding first image pixel value according to the first image data.

[0126] In the embodiment of the present application, after the first image data is acquired as described above, a pixel value corresponding to the first image data, that is, a first image pixel value, may be determined.

[0127] S503: Determine a second loss according to the third image pixel value and the first image pixel value.

[0128] In an embodiment of the present application, after the first image pixel value and the third image pixel value are obtained as described above, the mean square error between the first image pixel value and the third image pixel value can be determined, and the second loss can be determined based on the mean square error between the first image pixel value and the third image pixel value.

[0129] The method for obtaining the second loss provided in the present application determines the second loss by the difference between the pixel values ​​corresponding to the third image data and the first image data, thereby providing a data basis for subsequently determining the target loss and training the target classification model based on the target loss.

[0130] In one embodiment, see Figure 8 , and also provides a complete classification model training method, including:

[0131] S10, acquiring unlabeled first image data and labeled second image data;

[0132] S11, inputting the second image data into the first initial classification sub-model for training to obtain a first classification sub-model;

[0133] S12, inputting the first image data into the first classification sub-model for classification to obtain a second classification result;

[0134] S13, performing block processing on the first image data to obtain a plurality of image blocks;

[0135] S14, performing mask processing on some image blocks among the multiple image blocks to obtain a masked image;

[0136] S15, inputting the masked image into an encoder for encoding to obtain an encoded image;

[0137] S16, inputting the encoded image into a decoder for completion processing to obtain third image data;

[0138] S17, inputting the encoded image into a perceptron for classification to obtain a first classification result;

[0139] S18, determining a first loss according to the first classification result and the second classification result output by the first classification sub-model;

[0140] S19, determining a corresponding third image pixel value according to the third image data;

[0141] S20, determining a corresponding first image pixel value according to the first image data;

[0142] S21, determining a second loss according to the third image pixel value and the first image pixel value;

[0143] S22. According to the second loss and the first loss, the preset classification model is trained to obtain a target classification model.

[0144] The above method provides a method for training a classification model based on labeled images and unlabeled images, which solves the problem that the training of computer vision models is difficult in current eye disease recognition applications; in addition, the preset classification model is trained based on a first loss determined by a classification result corresponding to the preset classification model and a second loss determined based on image information corresponding to the preset classification model, thereby ensuring the accuracy of the classification model obtained after training.

[0145] In one embodiment, in order to ensure that in real environments, especially in the medical field, data generated by users are usually associated with or involve the privacy of users, directly centralizing a wide range of user data for model training often leads to the risk of privacy leakage. To solve this problem, the federated learning field proposes a method of using data from each party separately for training models and centralizing the trained models. This method formally solves the privacy problem caused by the centralization of data sets.

[0146] Therefore, this application also provides a decentralized federated learning privacy protection mechanism based on the MQTT protocol. The MQTT protocol adopts a publish-subscribe mechanism. The participants need to agree on an MQTT message server that is reachable by all participants) and a root topic. Then, federated learning can be performed according to the following steps, including:

[0147] 1) Sharing of model parameters among participants: Taking a single participant as an example, after each round of training, testing and parameter synchronization, some model training parameters are selected and published to the sub-topic of the corresponding parameter name under the root topic of the MQTT message server with network service quality QoS=0. After that, all participants who have subscribed to the root topic will receive the updated parameter values ​​and the corresponding parameter names.

[0148] 2) Receiving and synchronizing model parameters of other participants:

[0149] The federated learning method provided in this application does not require the training time of each participant to be completely synchronized. ), during the model training process, it is possible to receive model parameters, sample quantity and model performance information published by other participants. At this time, the participant will temporarily store this information without affecting the normal progress of its own training process. After the training and testing process of this round is completed, the participant will then And the performance of the test , and the relevant information received from other participants is calculated by the federated average algorithm to obtain the weighted average value, see formula (2):

[0150]

[0151] in, Known by the name The set of participants in the parameter update of For participants of The model parameters, The name obtained after weighted average is The model parameters.

[0152] Optionally, after obtaining the weighted average, the model training of the participant can be performed based on the weighted average.

[0153] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0154] Based on the same inventive concept, the embodiment of the present application also provides a classification model training device for implementing the classification model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more classification models provided below can refer to the limitations of the classification model training method above, and will not be repeated here.

[0155] In an exemplary embodiment, as shown in FIG9 , a classification model training device is provided, comprising: an acquisition module 10, an output module 11, a determination module 12 and a training module 13, wherein:

[0156] The acquisition module 10 is used to acquire the first image data without labels and the second image data with labels.

[0157] The output module 11 is used to input the first image data and the second image data into a preset classification model for training, and output the first classification result of the first image data, the first classification sub-model corresponding to the second image data, and the third image data; the third image data is the image after mask processing and completion processing.

[0158] The determination module 12 is used to determine the first loss according to the first classification result and the second classification result output by the first classification sub-model, and to determine the second loss according to the third image data and the first image data.

[0159] The training module 13 is used to train the preset classification model according to the second loss and the first loss to obtain a target classification model.

[0160] In an exemplary embodiment, the preset classification model includes a first initial classification sub-model and a second initial classification sub-model, and the output module 11 includes: a training unit, a classification unit and a processing unit, wherein:

[0161] A training unit, specifically used to input the second image data into the first initial classification sub-model for training to obtain a first classification sub-model;

[0162] A classification unit, specifically used to input the first image data into the first classification sub-model for classification, to obtain a second classification result;

[0163] The processing unit is specifically used to input the first image data into the second initial classification sub-model for processing to obtain the first classification result and the third image data.

[0164] In an exemplary embodiment, the second initial classification sub-model includes an encoder and a decoder, and the processing unit is further used to perform mask processing on the first image data to obtain a masked image; input the masked image into the encoder for encoding to obtain an encoded image; input the encoded image into the decoder for completion processing to obtain third image data.

[0165] In an exemplary embodiment, the second initial classification sub-model further includes a perceptron, and the processing unit is further configured to input the encoded image into the perceptron for classification to obtain a first classification result.

[0166] In an exemplary embodiment, the processing unit is further configured to perform block processing on the first image data to obtain a plurality of image blocks; and perform mask processing on some of the plurality of image blocks to obtain a masked image.

[0167] In an exemplary embodiment, the above-mentioned determination module 12 includes: a first determination unit, a second determination unit and a third determination unit, wherein:

[0168] A first determining unit, specifically configured to determine a corresponding third image pixel value according to the third image data;

[0169] A second determining unit, specifically configured to determine a corresponding first image pixel value according to the first image data;

[0170] The third determining unit is specifically configured to determine the second loss according to the third image pixel value and the first image pixel value.

[0171] Each module in the training device of the above classification model can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0172] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the first image data and the second image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a training method for a classification model is implemented.

[0173] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0174] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0175] Acquire unlabeled first image data and labeled second image data;

[0176] Input the first image data and the second image data into a preset classification model for training, and output a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing;

[0177] Determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and determine a second loss according to the third image data and the first image data;

[0178] According to the second loss and the first loss, the preset classification model is trained to obtain the target classification model.

[0179] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0180] Inputting the second image data into the first initial classification sub-model for training to obtain a first classification sub-model;

[0181] Inputting the first image data into the first classification sub-model for classification to obtain a second classification result;

[0182] The first image data is input into the second initial classification sub-model for processing to obtain a first classification result and third image data.

[0183] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0184] Performing mask processing on the first image data to obtain a masked image;

[0185] Inputting the masked image into an encoder for encoding to obtain an encoded image;

[0186] The encoded image is input into the decoder for completion processing to obtain third image data.

[0187] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0188] The encoded image is input into the perceptron for classification to obtain the first classification result.

[0189] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0190] Performing block processing on the first image data to obtain a plurality of image blocks;

[0191] Mask processing is performed on some image blocks among the multiple image blocks to obtain a masked image.

[0192] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0193] Determining a corresponding third image pixel value according to the third image data;

[0194] Determine a corresponding first image pixel value according to the first image data;

[0195] A second loss is determined based on the third image pixel value and the first image pixel value.

[0196] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0197] Acquire unlabeled first image data and labeled second image data;

[0198] Input the first image data and the second image data into a preset classification model for training, and output a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing;

[0199] Determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and determine a second loss according to the third image data and the first image data;

[0200] According to the second loss and the first loss, the preset classification model is trained to obtain the target classification model.

[0201] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0202] Inputting the second image data into the first initial classification sub-model for training to obtain a first classification sub-model;

[0203] Inputting the first image data into the first classification sub-model for classification to obtain a second classification result;

[0204] The first image data is input into the second initial classification sub-model for processing to obtain a first classification result and third image data.

[0205] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0206] Performing mask processing on the first image data to obtain a masked image;

[0207] Inputting the masked image into an encoder for encoding to obtain an encoded image;

[0208] The encoded image is input into the decoder for completion processing to obtain third image data.

[0209] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0210] The encoded image is input into the perceptron for classification to obtain the first classification result.

[0211] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0212] Performing block processing on the first image data to obtain a plurality of image blocks;

[0213] Mask processing is performed on some image blocks among the multiple image blocks to obtain a masked image.

[0214] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0215] Determining a corresponding third image pixel value according to the third image data;

[0216] Determine a corresponding first image pixel value according to the first image data;

[0217] A second loss is determined based on the third image pixel value and the first image pixel value.

[0218] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0219] Acquire unlabeled first image data and labeled second image data;

[0220] Input the first image data and the second image data into a preset classification model for training, and output a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing;

[0221] Determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and determine a second loss according to the third image data and the first image data;

[0222] According to the second loss and the first loss, the preset classification model is trained to obtain the target classification model.

[0223] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0224] Inputting the second image data into the first initial classification sub-model for training to obtain a first classification sub-model;

[0225] Inputting the first image data into the first classification sub-model for classification to obtain a second classification result;

[0226] The first image data is input into the second initial classification sub-model for processing to obtain a first classification result and third image data.

[0227] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0228] Performing mask processing on the first image data to obtain a masked image;

[0229] Inputting the masked image into an encoder for encoding to obtain an encoded image;

[0230] The encoded image is input into the decoder for completion processing to obtain third image data.

[0231] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0232] The encoded image is input into the perceptron for classification to obtain the first classification result.

[0233] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0234] Performing block processing on the first image data to obtain a plurality of image blocks;

[0235] Mask processing is performed on some image blocks among the multiple image blocks to obtain a masked image.

[0236] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0237] Determining a corresponding third image pixel value according to the third image data;

[0238] Determine a corresponding first image pixel value according to the first image data;

[0239] A second loss is determined based on the third image pixel value and the first image pixel value.

[0240] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0241] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0242] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0243] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A classification model training method, characterized in that: The method comprises: Acquire unlabeled first image data and labeled second image data; Input the first image data and the second image data into a preset classification model for training, and output a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing; Determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and determine a second loss according to the third image data and the first image data; The preset classification model is trained according to the second loss and the first loss to obtain a target classification model.

2. The method according to claim 1, characterized in that The preset classification model includes a first initial classification sub-model and a second initial classification sub-model, and the first image data and the second image data are input into the preset classification model for training, and a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data are output, including: Inputting the second image data into the first initial classification sub-model for training to obtain the first classification sub-model; Inputting the first image data into the first classification sub-model for classification to obtain the second classification result; The first image data is input into the second initial classification sub-model for processing to obtain the first classification result and the third image data.

3. The method according to claim 2, characterized in that The second initial classification sub-model includes an encoder and a decoder, and the first image data is input into the second initial classification sub-model for processing to obtain the third image data, including: Performing mask processing on the first image data to obtain a masked image; Inputting the masked image into the encoder for encoding to obtain an encoded image; The encoded image is input into the decoder for completion processing to obtain the third image data.

4. The method according to claim 3, characterized in that The second initial classification sub-model further includes a perceptron, and the first image data is input into the second initial classification sub-model for processing to obtain the first classification result, including: The encoded image is input into the perceptron for classification to obtain the first classification result.

5. The method according to claim 3, characterized in that: The performing mask processing on the first image data to obtain a masked image includes: Performing block processing on the first image data to obtain a plurality of image blocks; Mask processing is performed on some image blocks among the multiple image blocks to obtain the masked image.

6. The method according to any one of claims 1 to 5, characterized in that: The determining the second loss according to the third image data and the first image data comprises: Determining a corresponding third image pixel value according to the third image data; Determining a corresponding first image pixel value according to the first image data; The second loss is determined according to the third image pixel value and the first image pixel value.

7. A training device for a classification model, characterized in that: The device comprises: An acquisition module, used for acquiring unlabeled first image data and labeled second image data; a training module, configured to input the first image data and the second image data into a preset classification model for training, and output a first classification result of the first image data, a first classification sub-model corresponding to the second image data, and third image data; the third image data is an image after mask processing and completion processing; a determination module, configured to determine a first loss according to the first classification result and a second classification result output by the first classification sub-model, and to determine a second loss according to the third image data and the first image data; A training module is used to train the preset classification model according to the second loss and the first loss to obtain a target classification model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images