Image classification method and related equipment for incremental learning

Abstract

The present application relates to the fields of artificial intelligence and digital medicine, and provides an image classification method and related equipment for incremental learning. The method determines the classification task type of a first local image sample set, and obtains corresponding multiple adapters and prediction output modules according to the target classification task type. Then, the multiple adapters, prediction output modules and pre-trained models are combined to obtain an initial image classification model. Then, federated learning is performed based on the initial image classification model to obtain an intermediate image classification model, and the intermediate image classification model is used as a teacher model to distill the initial image classification model to obtain a target image classification model. Finally, the target image classification model is used to perform classification prediction. This method can not only fully learn new types of data under the federated learning framework, but also alleviate the catastrophic forgetting caused by learning new types of data, thereby improving the incremental learning ability of image classification under the federated learning architecture.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and digital medicine, and in particular to an image classification method for incremental learning and related equipment. Background Art

[0002] With the development of artificial intelligence, deep learning technologies are increasingly being applied across various industries. However, the need for data privacy protection in image classification applications within the computer vision field prevents the sharing of raw data, resulting in the formation of "data silos" between multiple data sources. Therefore, image classification based on federated learning has been proposed. By training classification models through federated learning, each federated learning participant can benefit from the image data of other participants while ensuring that each participant's image data remains locally. This successfully resolves the data silo issue while ensuring the data privacy of all parties.

[0003] For example, in the field of digital medicine, using deep learning technology to process medical images is a crucial step in medical-assisted diagnosis. Training deep learning-based classification models through federated learning methods can solve the data silo problem while protecting medical data privacy and improve the accuracy of medical image classification.

[0004] Furthermore, practical image classification applications also require models to possess incremental learning capabilities, meaning they can continuously learn new knowledge from new data without forgetting previously learned knowledge. However, most current federated learning-based image classification methods suffer from catastrophic forgetting during incremental learning. This means that after learning new categories, the global model's performance on old categories significantly degrades.

[0005] Therefore, how to improve the incremental learning capability of image classification under the federated learning architecture has become a technical problem that needs to be solved urgently. Summary of the Invention

[0006] The main purpose of the embodiments of the present application is to propose an image classification method, device, electronic device and computer-readable storage medium for incremental learning, which can improve the incremental learning ability of image classification under the federated learning architecture and alleviate the catastrophic forgetting phenomenon.

[0007] To achieve the above objectives, a first aspect of an embodiment of the present application proposes an image classification method for incremental learning, the method comprising:

[0008] Acquire a first local image sample set;

[0009] determining a target classification task type according to the first local image sample set;

[0010] Acquire multiple adapters and prediction output modules corresponding to the target classification task type;

[0011] Combining the plurality of adapters, the prediction output modules, and the pre-trained model to obtain an initial image classification model;

[0012] Training the initial image classification model based on the first local image sample set to obtain an intermediate image classification model;

[0013] Uploading the local model parameters of the intermediate image classification model to the server, so that the server integrates the local model parameters to obtain global model parameters;

[0014] Obtaining the global model parameters from the server, and updating the intermediate image classification model according to the global model parameters;

[0015] Using the updated intermediate image classification model as a teacher model to perform distillation processing on the initial image classification model to obtain a target image classification model;

[0016] Get the image to be classified;

[0017] The image to be classified is input into the target image classification model to obtain a classification prediction result corresponding to the image to be classified through the target image classification model.

[0018] According to some embodiments of the present invention, the image classification method for incremental learning is provided, wherein the updated intermediate image classification model is used as a teacher model to perform distillation processing on the initial image classification model to obtain a target image classification model, including:

[0019] Acquire a second local image sample set, where the second local image sample set and the first local image sample set are old image samples of the same classification task type;

[0020] fusing the first local image sample set and the second local image sample set to obtain a training sample set;

[0021] Inputting the training sample sets into the initial image classification model respectively, so as to obtain first classification prediction results of the image samples in the training sample sets through the initial image classification model;

[0022] Inputting the training sample sets into the updated intermediate image classification model respectively, so as to obtain second classification prediction results of the image samples in the training sample set through the intermediate image classification model;

[0023] Determine a loss value based on the first classification prediction result and the second classification prediction result;

[0024] Based on the loss value, model parameters of multiple adapters and the prediction output module in the initial image classification model are updated to obtain a target image classification model.

[0025] According to the incremental learning-oriented image classification method provided by some embodiments of the present invention, before obtaining the plurality of adapters and prediction output modules corresponding to the target classification task type, the method further includes:

[0026] Constructing a network module pool, wherein the network module pool includes a plurality of network module groups, and the network module groups are used to be combined with a pre-trained model to obtain an image classification model;

[0027] Each of the network module groups corresponds to a classification task type, and each of the network module groups includes multiple adapters and prediction output modules;

[0028] The step of obtaining multiple adapters and prediction output modules corresponding to the target classification task type includes:

[0029] According to the correspondence between the classification task type and the network module group, a plurality of adapters and a prediction output module corresponding to the target classification task type are obtained from the network module pool.

[0030] According to the image classification method for incremental learning provided by some embodiments of the present invention, after performing distillation processing on the initial image classification model using the updated intermediate image classification model as a teacher model to obtain a target image classification model, the method further includes:

[0031] Obtaining first model parameters of a plurality of the adapters and the prediction output modules in the target image classification model;

[0032] The corresponding network module group in the network module pool is updated according to the first model parameter.

[0033] According to the incremental learning-oriented image classification method provided by some embodiments of the present invention, after obtaining first model parameters of the plurality of adapters and the prediction output module in the target image classification model, the method further includes:

[0034] Uploading the first model parameters to the server so that the server integrates the first model parameters to obtain second model parameters;

[0035] The updating of the corresponding network module group in the network module pool according to the first model parameter includes:

[0036] Acquire second model parameters from the server, and update the corresponding network module group in the network module pool according to the second model parameters.

[0037] According to the incremental learning-oriented image classification method provided by some embodiments of the present invention, determining the target classification task type based on the first local image sample set includes:

[0038] Obtain the trained first image classification model;

[0039] Selecting a preset number of image samples from the first local image sample set as a test sample set;

[0040] Inputting the test sample set into the first image classification model to obtain a classification prediction result corresponding to the test sample set through the first image classification model;

[0041] According to the classification prediction result, a classification task type corresponding to the first local image sample set is determined.

[0042] According to the image classification method for incremental learning provided by some embodiments of the present invention, the global model parameters are obtained by the following formula:

[0043]

[0044] Among them, the is the global model parameter, is the number of clients participating in federated learning. is the number of local image samples uploaded by the kth client. The local model parameters uploaded by the k-th client.

[0045] To achieve the above-mentioned objectives, a second aspect of an embodiment of the present application provides an image classification device for incremental learning, the device comprising:

[0046] A first acquisition module, configured to acquire a first local image sample set;

[0047] a classification module, configured to determine a target classification task type based on the first local image sample set;

[0048] A second acquisition module is used to acquire multiple adapters and prediction output modules corresponding to the target classification task type;

[0049] A model combining module, configured to combine the plurality of adapters, the prediction output modules, and the pre-trained model to obtain an initial image classification model;

[0050] A model training module, configured to train the initial image classification model based on the first local image sample set to obtain an intermediate image classification model;

[0051] a parameter uploading module, configured to upload the local model parameters of the intermediate image classification model to a server, so that the server integrates the local model parameters to obtain global model parameters;

[0052] a third acquisition module, configured to acquire the global model parameters from the server and update the intermediate image classification model according to the global model parameters;

[0053] a model distillation module, configured to perform distillation processing on the initial image classification model using the updated intermediate image classification model as a teacher model to obtain a target image classification model;

[0054] A fourth acquisition module is used to acquire an image to be classified;

[0055] The image classification module is used to input the image to be classified into the target image classification model to obtain a classification prediction result corresponding to the image to be classified through the target image classification model.

[0056] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program implements the method described in the first aspect above when executed by the processor.

[0057] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the method described in the first aspect above.

[0058] The present application proposes an image classification method, device, electronic device, and computer-readable storage medium for incremental learning. The method first determines the classification task type of a first local image sample set, and obtains corresponding multiple adapters and prediction output modules according to the target classification task type. Then, the multiple adapters and prediction output modules are combined with a pre-trained model to obtain an initial image classification model. The initial image classification model is trained based on the first local image sample set to obtain an intermediate image classification model. The local model parameters of the intermediate image classification model are then uploaded to a server so that the server integrates the local model parameters to obtain global model parameters. The global model parameters are obtained from the server, and the intermediate image classification model is updated according to the global model parameters. The updated intermediate image classification model is used as a teacher model to perform distillation on the initial image classification model to obtain a target image classification model. Finally, the target image classification model is used to perform image classification. This method can fully learn new class data under the federated learning framework and use the learned new model for image classification. It can also alleviate catastrophic forgetting caused by learning new class data and improve the incremental learning ability of image classification under the federated learning architecture.

[0059] An image classification method for incremental learning proposed in an embodiment of the present application can be applied to the field of digital medicine, such as a medical auxiliary diagnosis system based on artificial intelligence technology. The medical auxiliary diagnosis system includes a medical terminal device and a medical cloud server. In a medical application scenario, the medical terminal device first determines the target classification task type of a first local medical image sample set, and obtains corresponding multiple adapters and prediction output modules according to the target classification task type. Then, the multiple adapters and prediction output modules are combined with the pre-trained model to obtain an initial image classification model. The initial image classification model is then trained based on the first local medical image sample set, and the local model parameters of the intermediate image classification model are uploaded to the medical cloud in the medical auxiliary diagnosis system. The server integrates the local model parameters uploaded by multiple medical terminal devices to obtain global model parameters. The medical terminal devices then obtain the global model parameters from the medical cloud server and update the intermediate image classification model according to the global model parameters. The updated intermediate image splitting model is used as the teacher model to distill the initial image classification model to obtain the target image classification model. Finally, the medical terminal devices use the target image classification model to classify the medical images. This can fully learn new types of data while protecting the privacy of medical data, and alleviate the catastrophic forgetting caused by learning new types of data through the knowledge distillation method, providing the medical terminal devices with incremental learning capabilities when facing image classification, thereby improving the accuracy of medical image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1This is a flowchart of an image classification method for incremental learning provided by an embodiment of the present application;

[0061] Figure 2 yes Figure 1 Schematic diagram of the sub-step flow of step S170;

[0062] Figure 3 yes Figure 1 Schematic diagram of the sub-step flow of step S120;

[0063] Figure 4 This is a flowchart of an image classification method for incremental learning provided by another embodiment of the present application;

[0064] Figure 5 This is a flowchart of an image classification method for incremental learning provided by another embodiment of the present application;

[0065] Figure 6 yes Figure 1 Schematic diagram of the sub-step flow of step S110;

[0066] Figure 7 This is a schematic diagram of the structure of a federated learning framework provided in an embodiment of the present application;

[0067] Figure 8 This is a flowchart of an image classification method for incremental learning provided by another embodiment of the present application;

[0068] Figure 9 This is a schematic structural diagram of an image classification device for incremental learning provided by another embodiment of the present application;

[0069] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0071] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0072] With the development of artificial intelligence, deep learning technologies are increasingly being applied across various industries. However, the need for data privacy protection in image classification applications within the computer vision field prevents the sharing of raw data, resulting in the formation of "data silos" between multiple data sources. Therefore, image classification based on federated learning has been proposed. By training classification models through federated learning, each federated learning participant can benefit from the image data of other participants while ensuring that each participant's image data remains locally. This successfully resolves the data silo issue while ensuring the data privacy of all parties.

[0073] For example, in the field of digital medicine, using deep learning technology to process medical images is a crucial step in medical-assisted diagnosis. Training deep learning-based classification models through federated learning methods can solve the data silo problem while protecting medical data privacy and improve the accuracy of medical image classification.

[0074] Furthermore, practical image classification applications also require models to possess incremental learning capabilities, meaning they can continuously learn new knowledge from new data without forgetting previously learned knowledge. However, most current federated learning-based image classification methods suffer from catastrophic forgetting during incremental learning. This means that after learning new categories, the global model's performance on old categories significantly degrades.

[0075] Therefore, how to improve the incremental learning capability of image classification under the federated learning architecture has become a technical problem that needs to be solved urgently.

[0076] Based on this, the embodiments of the present application provide an image classification method, device, electronic device and computer-readable storage medium for incremental learning, which can improve the incremental learning ability of image classification under the federated learning architecture and alleviate the catastrophic forgetting phenomenon.

[0077] The embodiments of the present application provide an image classification method, device, electronic device, and computer-readable storage medium for incremental learning, which are specifically illustrated by the following embodiments. First, the image classification method for incremental learning in the embodiments of the present application is described.

[0078] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0079] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0080] The image classification method for incremental learning provided in the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image classification method for incremental learning, etc., but is not limited to the above forms.

[0081] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0082] First, the federated learning framework in the embodiment of this application is described, see Figure 7 , Figure 7 This is a schematic diagram of the structure of a federated learning framework provided in an embodiment of the present application. Figure 7 As shown, the federated learning framework includes a server and multiple clients, and the server and the multiple clients are communicated with each other.

[0083] See Figure 1 , Figure 1 FIG. 1 shows a flow chart of an image classification method for incremental learning provided by an embodiment of the present application. Figure 1 As shown, the image classification method for incremental learning is applied to a client participating in federated learning, and the method includes but is not limited to steps S100 to S190:

[0084] Step S100, obtaining a first local image sample set;

[0085] Step S110, determining a target classification task type according to the first local image sample set;

[0086] Step S120, obtaining multiple adapters and prediction output modules corresponding to the target classification task type;

[0087] Step S130, combining the plurality of adapters, the prediction output modules and the pre-trained model to obtain an initial image classification model;

[0088] Step S140, training the initial image classification model based on the first local image sample set to obtain an intermediate image classification model;

[0089] Step S150, uploading the local model parameters of the intermediate image classification model to the server, so that the server integrates the local model parameters to obtain global model parameters;

[0090] Step S160, obtaining the global model parameters from the server, and updating the intermediate image classification model according to the global model parameters;

[0091] Step S170, using the updated intermediate image classification model as a teacher model to perform distillation processing on the initial image classification model to obtain a target image classification model;

[0092] Step S180, obtaining an image to be classified;

[0093] Step S190: input the image to be classified into the target image classification model to obtain a classification prediction result corresponding to the image to be classified through the target image classification model.

[0094] It is understandable that when a client participating in federated learning learns a new task, it obtains a first local image sample set and determines the corresponding target classification task type based on the first local image sample set, that is, based on the new class data.

[0095] In one possible implementation, the image classification method for incremental learning provided in the embodiments of the present application is applied to the field of digital medicine, such as a medical auxiliary diagnosis system based on artificial intelligence technology, wherein the medical auxiliary diagnosis system includes multiple medical terminal devices and a medical cloud server, wherein the medical terminal devices and the medical cloud server are connected via wired or wireless communication and jointly participate in federated learning, and the medical terminal devices execute the image classification method for incremental learning, wherein, in a medical application scenario, the sample image is a medical image, and the type of object contained in the sample image is a lesion, that is, a part of the body where a lesion occurs. Medical images refer to images of internal tissues obtained in a non-invasive manner for medical treatment or medical research, such as images of the stomach, abdomen, heart, knee, and brain, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), US (ultrasonic), X-ray images, electroencephalograms, and optical photography, etc., which are generated by medical instruments.

[0096] In some embodiments, see Figure 6 , Figure 6 yes Figure 1 The sub-step flow diagram of step S110 is as follows: Figure 6 As shown, determining the target classification task type according to the first local image sample set includes:

[0097] Step S610, obtaining a trained first image classification model;

[0098] Step S620: selecting a preset number of image samples from the first local image sample set as a test sample set;

[0099] Step S630: input the test sample set into the first image classification model to obtain a classification prediction result corresponding to the test sample set through the first image classification model;

[0100] Step S640: Determine the classification task type corresponding to the first local image sample set according to the classification prediction result.

[0101] In one specific embodiment, the first image classification model is an EfficientNet model. A pre-set number of image samples from the first local image sample set are selected as a test sample set by obtaining a trained EfficientNet model. The test sample set is then input into the EfficientNet model to obtain a classification prediction result corresponding to the test sample set. The classification task type corresponding to the first local image sample set is then determined based on the classification prediction result. In other words, when a new class needs to be learned, the image sample set is first roughly classified using EfficientNet to determine the classification task type to which the new class belongs.

[0102] After the target classification task type, multiple adapters and prediction output modules corresponding to the target classification task type are obtained.

[0103] In some embodiments, before obtaining a plurality of adapters and prediction output modules corresponding to the target classification task type, the method further includes:

[0104] Constructing a network module pool, the network module pool including a plurality of network module groups, the network module groups being used to be combined with a pre-trained model to obtain an image classification model;

[0105] Each of the network module groups corresponds to a classification task type, and each of the network module groups includes multiple adapters and prediction output modules;

[0106] It is understandable that before obtaining multiple adapters and prediction output modules corresponding to the target classification task type, a network module pool is pre-built, wherein, see Figure 8 The network module pool includes multiple network module groups, and the network module group includes multiple adapters (Adapter) and prediction output modules (Prediction head). At the same time, each network module group corresponds to a classification task type.

[0107] See Figure 3 , Figure 3 yes Figure 1 The sub-step flow diagram of step S120 is as follows: Figure 3 As shown, multiple adapters and prediction output modules corresponding to the target classification task type are obtained, including but not limited to:

[0108] Step S310 : According to the correspondence between the classification task type and the network module group, a plurality of adapters and prediction output modules corresponding to the target classification task type are obtained from the network module pool.

[0109] It can be understood that since each network module group corresponds to a classification task type, after determining the target classification task type, according to the correspondence between the preset classification task type and the network module group, the network module group corresponding to the target classification task type can be obtained from the network module pool, that is, multiple adapters and prediction output modules corresponding to the target classification task type are determined.

[0110] It should be understood that the present application divides the classification task into multiple subcategories, and each subcategory has its corresponding multiple adapters and prediction output modules in the network module pool.

[0111] In a specific embodiment, the pre-trained model includes multiple Transformer blocks, such as Figure 8 As shown in the figure, after determining the target classification task type, multiple adapters and prediction output modules corresponding to the target classification task type are obtained, and then combined with the pre-trained Transformer blocks to obtain the initial image classification model.

[0112] It should be understood that by adding some modules with trainable parameters, namely adapter modules, to each Transformer block in the original pre-trained model, when adjusting for different downstream tasks, it is possible to train only the parameters of the adapter module while keeping the pre-trained parameters fixed. This can significantly reduce the communication cost of federated learning when using large-scale models.

[0113] After an initial image classification model is obtained by combining multiple adapters, prediction output modules, and pre-trained models, the initial image classification model is trained based on the first local image sample set to obtain an intermediate image classification model. The training process of the initial image classification model may include the following steps:

[0114] The first local image sample set is input into the initial image classification model to obtain the prediction results corresponding to the image samples in the first local image sample set through the initial image classification model, and the loss value is determined according to the prediction results, the true classification of the image and the preset classification loss function, and the initial image classification model is trained based on the loss value, that is, the model parameters of the initial image classification model are updated until the loss value meets the preset threshold, thereby obtaining an intermediate image classification model.

[0115] It should be understood that after the intermediate image classification model is obtained, the local model parameters of the intermediate image classification model are uploaded to the server so that the local model parameters are processed by the server to obtain the global model parameters.

[0116] It should be noted that the local model parameters of the intermediate image classification model refer to the parameters of the adapter. The image classification model of this application uses a model structure of a pre-trained model + adapter. After combining multiple adapters, prediction output modules, and pre-trained models to obtain an initial image classification model, the initial image classification model is trained. That is, when the model parameters of the pre-trained model are fixed, only the parameters of the adapter are trained. Therefore, in the process of integrating the model parameters, the client only needs to upload the parameters of the adapter in the intermediate image classification model to the server, and the server will integrate the parameters of the adapters uploaded by multiple clients to obtain the global model parameters.

[0117] In some embodiments, the global model parameters are obtained by the following formula:

[0118]

[0119] Among them, the is a global model parameter. is the number of clients participating in federated learning. is the number of local image samples uploaded by the k-th client. The local model parameters uploaded by the k-th client.

[0120] For example, if Figure 7 As shown in the figure, N clients participating in federated learning upload their respective local model parameters and the number of local image samples. The server calculates according to the above formula to obtain the global model parameters of this federated learning.

[0121] In some other embodiments, after uploading the local model parameters of the intermediate image classification model to the server, the method further includes:

[0122] Get the response information sent by the server,

[0123] The step of training the initial image classification model based on the first local image sample set is returned according to the response information.

[0124] It can be understood that the server and the local client are also preset with a network module pool. After the server integrates the local model parameters uploaded by multiple clients to obtain the global model parameters, it updates the image classification model composed of multiple adapters, prediction output modules and pre-trained models in the server based on the global model parameters, and tests based on the image classification model; if the test results meet the preset conditions, the model parameters obtained in this integration are regarded as the final global model parameters, so that the client obtains the global model parameters from the server and updates the intermediate image classification model according to the global model parameters; or, if the test results do not meet the preset conditions, a response message is sent to the client participating in this federated learning, so that the client continues to train the initial image classification model according to the response message until the test results meet the preset conditions.

[0125] By integrating the local model parameters uploaded by the client and testing the image classification model updated based on the global model parameters, determining whether to continue training the initial local image classification model based on the test results, it is possible to effectively avoid the problem of model performance degradation caused by non-independent and identically distributed data between clients.

[0126] In some embodiments, see Figure 2 , Figure 2 yes Figure 1 The sub-step flow chart of step S170 is as follows: Figure 2 As shown, the updated intermediate image classification model is used as a teacher model to perform distillation on the initial image classification model to obtain a target image classification model, including:

[0127] Step S210: Acquire a second local image sample set, where the second local image sample set and the first local image sample set are old image samples of the same classification task type;

[0128] Step S220, fusing the first local image sample set and the second local image sample set to obtain a training sample set;

[0129] Step S230, inputting the training sample set into the initial image classification model respectively, so as to obtain a first classification prediction result of the image samples in the training sample set through the initial image classification model;

[0130] Step S240: inputting the training sample set into the updated intermediate image classification model respectively, so as to obtain second classification prediction results of the image samples in the training sample set through the intermediate image classification model;

[0131] Step S250, determining a loss value according to the first classification prediction result and the second classification prediction result;

[0132] Step S260: Based on the loss value, the model parameters of the multiple adapters and the prediction output module in the initial image classification model are updated to obtain a target image classification model.

[0133] It should be understood that the first local image sample set and the second local image sample set are merged to obtain a training sample set, that is, the sample data of the new and old tasks are fused, and then the training sample set is used to perform knowledge distillation on the initial image classification model and the intermediate image classification model, wherein the intermediate image classification model serves as the teacher model and the initial image classification model serves as the student model.

[0134] By performing knowledge distillation on the intermediate image classification model after learning new class data to update the model parameters of the initial image classification model, the local model in the client can fully learn the new class data and use it to perform image classification, while also alleviating catastrophic forgetting caused by learning new class data.

[0135] In some embodiments, see Figure 4 , Figure 4 Another embodiment of the present application provides an image classification method for incremental learning, such as Figure 4 As shown, after the updated intermediate image classification model is used as a teacher model to perform distillation processing on the initial image classification model to obtain a target image classification model, the method further includes:

[0136] Step S410, obtaining first model parameters of a plurality of the adapters and the prediction output modules in the target image classification model;

[0137] Step S420: Update the corresponding network module group in the network module pool according to the first model parameter.

[0138] It should be understood that after performing knowledge distillation on the intermediate image classification model and the initial image classification model to update the model parameters of the initial image classification model, the parameters of multiple adapters and prediction output models in the target image classification model are obtained to update the corresponding network module group in the network module pool, that is, replace the parameters of the adapter and prediction output module in the network module group corresponding to the target classification task type, so that the client can obtain the network module group that has been learned multiple times from the network module pool in the next sample learning, thereby realizing the client's federated incremental learning.

[0139] In some embodiments, see Figure 5 , Figure 5 Another embodiment of the present application provides an image classification method for incremental learning, such as Figure 5 As shown, after obtaining the first model parameters of the plurality of adapters and the prediction output module in the target image classification model, the method further includes:

[0140] Step S510: uploading the first model parameters to the server so that the server integrates the first model parameters to obtain second model parameters;

[0141] The updating of the corresponding network module group in the network module pool according to the first model parameter includes:

[0142] Step S520: Obtain second model parameters from the server, and update the corresponding network module group in the network module pool according to the second model parameters.

[0143] It should be understood that for the first model parameters of multiple adapters and prediction output modules in the target image classification model, before updating the corresponding network module groups in the network module pool according to the first model parameters, the first model parameters can be uploaded to the server, so that the server can process the first model parameters uploaded by multiple clients to obtain the second model parameters, so that the clients can update the corresponding network module groups in the network module pool according to the second model parameters, which can effectively avoid the problem of non-independent and identically distributed data between clients.

[0144] It is also understandable that the image classification model composed of multiple adapters, prediction output modules, and pre-trained models also includes an embedding block, which is used to extract features from the model input.

[0145] The following describes the image classification method for incremental learning provided by the embodiment of the present application through a specific embodiment:

[0146] like Figure 7 and 8 As shown, the method is applied to a client participating in federated learning, which is in communication with a server. A network module pool is pre-built in the client, where the network module pool includes N network module groups, each of which includes multiple adapters and prediction output modules, and each network module group corresponds to a subclass in the classification task.

[0147] The client obtains a first local image sample set and uses it to learn new knowledge. The client first uses a preset EfficientNet model to perform coarse classification on the first local image sample set to determine the target classification task type corresponding to the first local image sample set. Then, based on the correspondence between the preset network module group and the classification task type, the client determines multiple adapters and prediction output modules corresponding to the target classification task type. The client then combines the multiple adapters and prediction output modules with the pre-trained Transformer block to obtain an initial image classification model.

[0148] Then, federated learning is performed on the initial image classification model to obtain an intermediate image classification model. The intermediate image classification model is then used as a teacher model to perform distillation on the initial image classification model to obtain a target image classification model. Finally, classification prediction is performed based on the target image classification model. At the same time, the first model parameters of multiple adapters and prediction output modules in the target image classification model are obtained, and the corresponding network module group in the network module pool is updated according to the first model parameters, thereby fully learning new types of data under the federated learning framework and using the learned target image classification model for classification prediction. At the same time, it can alleviate catastrophic forgetting caused by learning new types of data and effectively avoid the problem of model performance degradation caused by non-independent and identically distributed data between clients.

[0149] The following describes the application scenarios of the embodiments of the present application through specific examples:

[0150] The image classification method for incremental learning provided in the embodiment of the present application is applied to the field of digital medicine, such as a medical auxiliary diagnosis system based on artificial intelligence technology, which includes multiple medical terminal devices, a medical cloud server and a database server, wherein the medical terminal devices and the medical cloud server are connected through wired or wireless communications and jointly participate in federated learning, the medical terminal devices execute the image classification method for incremental learning, and a network module pool is pre-built in the database server, wherein the network module pool includes multiple network module groups, and each network module group includes multiple adapters and prediction output modules, and each network module group corresponds to a subclass in the classification task.

[0151] The medical terminal device obtains a first local medical image set and uses it to learn new knowledge. The medical terminal device first uses a preset EfficientNet model to roughly classify the first local medical image set to determine the target classification task type corresponding to the first local medical image set. The medical terminal device then determines multiple adapters and prediction output modules corresponding to the target classification task type from the network module group pre-built in the database server. The medical terminal device then combines the multiple adapters and prediction output modules with the pre-trained Transformer block to obtain an initial image classification model.

[0152] The medical terminal device then uploads the local model parameters of the initial image classification model to the medical cloud server. The medical cloud server integrates the local model parameters uploaded by multiple medical terminal devices to obtain global model parameters. The medical terminal device then obtains the global model parameters from the medical cloud server and updates the intermediate image classification model according to the global model parameters. Finally, the updated intermediate image classification model is used as the teacher model to perform distillation processing on the initial image classification model as the student model to obtain the target image classification model. Finally, the medical terminal device uses the target image classification model to perform image classification processing on the medical image. It can fully learn new types of data while protecting the privacy of medical data, and alleviate the catastrophic forgetting caused by learning new types of data through knowledge distillation methods, providing medical terminal devices with incremental learning capabilities when facing image classification, and improving the accuracy of medical image classification.

[0153] The present application proposes an image classification method for incremental learning. The method first determines the classification task type of the first local image sample set, and obtains corresponding multiple adapters and prediction output modules according to the target classification task type. Then, the multiple adapters and prediction output modules are combined with the pre-trained model to obtain an initial image classification model. The initial image classification model is trained based on the first local image sample set to obtain an intermediate image classification model. The local model parameters of the intermediate image classification model are then uploaded to the server so that the server integrates the local model parameters to obtain global model parameters. The global model parameters are obtained from the server, and the intermediate image classification model is updated according to the global model parameters. The updated intermediate image classification model is used as a teacher model to distill the initial image classification model to obtain a target image classification model. Finally, the target image classification model is used for image classification. This method can fully learn new class data under the federated learning framework and use the learned new model for image classification. It can also alleviate the catastrophic forgetting caused by learning new class data and improve the incremental learning ability of image classification under the federated learning architecture.

[0154] See Figure 9 , an embodiment of the present application further provides an image classification device for incremental learning, the image classification device for incremental learning comprising:

[0155] A first acquisition module 100 is configured to acquire a first local image sample set;

[0156] A classification module 110, configured to determine a target classification task type based on the first local image sample set;

[0157] A second acquisition module 120 is used to acquire multiple adapters and prediction output modules corresponding to the target classification task type;

[0158] A model combining module 130 is used to combine the plurality of adapters, the prediction output modules and the pre-trained model to obtain an initial image classification model;

[0159] A model training module 140 is configured to train the initial image classification model based on the first local image sample set to obtain an intermediate image classification model;

[0160] A parameter uploading module 150 is used to upload the local model parameters of the intermediate image classification model to the server so that the server integrates the local model parameters to obtain global model parameters;

[0161] A third acquisition module 160 is configured to acquire the global model parameters from the server and update the intermediate image classification model according to the global model parameters;

[0162] A model distillation module 170 is configured to distill the initial image classification model using the updated intermediate image classification model as a teacher model to obtain a target image classification model;

[0163] The fourth acquisition module 180 is used to acquire the image to be classified;

[0164] The image classification module 190 is used to input the image to be classified into the target image classification model to obtain a classification prediction result corresponding to the image to be classified through the target image classification model.

[0165] The present application proposes an image classification device for incremental learning, which first determines the classification task type of the first local image sample set, and obtains corresponding multiple adapters and prediction output modules according to the target classification task type, and then combines the multiple adapters and prediction output modules with the pre-trained model to obtain an initial image classification model. The initial image classification model is trained based on the first local image sample set to obtain an intermediate image classification model, and then the local model parameters of the intermediate image classification model are uploaded to the server so that the server integrates the local model parameters to obtain global model parameters, and the global model parameters are obtained from the server. The intermediate image classification model is updated according to the global model parameters, and the updated intermediate image classification model is used as a teacher model to distill the initial image classification model to obtain a target image classification model. Finally, the target image classification model is used for image classification. This device can fully learn new class data under the federated learning framework and use the learned new model for image classification, and can also alleviate the catastrophic forgetting caused by learning new class data, thereby improving the incremental learning ability of image classification under the federated learning architecture.

[0166] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned device are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0167] See Figure 10 , Figure 10 The hardware structure of an electronic device provided in an embodiment of the present application is shown. The electronic device includes:

[0168] The processor 210 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant computer programs to implement the technical solutions provided in the embodiments of the present application.

[0169] The memory 220 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 220 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called by the processor 210 to execute the image classification method for incremental learning in the embodiments of this application.

[0170] Input / output interface 230, used to implement information input and output;

[0171] Communication interface 240, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); and bus 250, which transmits information between each component of the device (such as processor 210, memory 220, input / output interface 230 and communication interface 240);

[0172] The processor 210 , the memory 220 , the input / output interface 230 and the communication interface 240 are connected to each other in communication within the device via the bus 250 .

[0173] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the above-mentioned image classification method for incremental learning.

[0174] The memory is a computer-readable storage medium that can be used to store software programs and computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0177] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0178] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0179] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0180] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or can be integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, the functional units in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0184] If the assembled units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0185] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. An image classification method for incremental learning, characterized in that: The method is applied to a client participating in federated learning, where the client is in communication with a server, and is characterized in that the method includes: Acquire a first local image sample set; determining a target classification task type according to the first local image sample set; Acquire multiple adapters and prediction output modules corresponding to the target classification task type; Combining the plurality of adapters, the prediction output modules, and the pre-trained model to obtain an initial image classification model; Training the initial image classification model based on the first local image sample set to obtain an intermediate image classification model; Uploading the local model parameters of the intermediate image classification model to the server, so that the server integrates the local model parameters to obtain global model parameters; Obtaining the global model parameters from the server, and updating the intermediate image classification model according to the global model parameters; Using the updated intermediate image classification model as a teacher model to perform distillation processing on the initial image classification model to obtain a target image classification model; Get the image to be classified; The image to be classified is input into the target image classification model to obtain a classification prediction result corresponding to the image to be classified through the target image classification model.

2. The method according to claim 1, characterized in that The step of using the updated intermediate image classification model as a teacher model to perform distillation processing on the initial image classification model to obtain a target image classification model includes: Acquire a second local image sample set, where the second local image sample set and the first local image sample set are old image samples of the same classification task type; fusing the first local image sample set and the second local image sample set to obtain a training sample set; Inputting the training sample sets into the initial image classification model respectively, so as to obtain first classification prediction results of the image samples in the training sample sets through the initial image classification model; Inputting the training sample sets into the updated intermediate image classification model respectively, so as to obtain second classification prediction results of the image samples in the training sample set through the intermediate image classification model; Determine a loss value based on the first classification prediction result and the second classification prediction result; Based on the loss value, model parameters of multiple adapters and the prediction output module in the initial image classification model are updated to obtain a target image classification model.

3. The method according to claim 1, characterized in that Before obtaining a plurality of adapters and prediction output modules corresponding to the target classification task type, the method further includes: Constructing a network module pool, wherein the network module pool includes a plurality of network module groups, and the network module groups are used to be combined with a pre-trained model to obtain an image classification model; Each of the network module groups corresponds to a classification task type, and each of the network module groups includes multiple adapters and prediction output modules; The step of obtaining multiple adapters and prediction output modules corresponding to the target classification task type includes: According to the correspondence between the classification task type and the network module group, a plurality of adapters and a prediction output module corresponding to the target classification task type are obtained from the network module pool.

4. The method according to claim 3, characterized in that After performing distillation processing on the initial image classification model using the updated intermediate image classification model as a teacher model to obtain a target image classification model, the method further includes: Obtaining first model parameters of a plurality of the adapters and the prediction output modules in the target image classification model; The corresponding network module group in the network module pool is updated according to the first model parameter.

5. The method according to claim 4, characterized in that After obtaining first model parameters of the plurality of adapters and the prediction output module in the target image classification model, the method further includes: Uploading the first model parameters to the server so that the server integrates the first model parameters to obtain second model parameters; The updating of the corresponding network module group in the network module pool according to the first model parameter includes: Acquire second model parameters from the server, and update the corresponding network module group in the network module pool according to the second model parameters.

6. The method according to claim 1, wherein The determining the target classification task type according to the first local image sample set includes: Obtain the trained first image classification model; Selecting a preset number of image samples from the first local image sample set as a test sample set; Inputting the test sample set into the first image classification model to obtain a classification prediction result corresponding to the test sample set through the first image classification model; According to the classification prediction result, a classification task type corresponding to the first local image sample set is determined.

7. The method according to claim 1, characterized in that The global model parameters are obtained by the following formula: Among them, the is the global model parameter, is the number of clients participating in federated learning. is the number of local image samples uploaded by the kth client. The local model parameters uploaded by the k-th client.

8. An image classification device for incremental learning, applied to a client participating in federated learning, characterized in that: The device comprises: A first acquisition module, configured to acquire a first local image sample set; a classification module, configured to determine a target classification task type based on the first local image sample set; A second acquisition module is used to acquire multiple adapters and prediction output modules corresponding to the target classification task type; A model combining module, configured to combine the plurality of adapters, the prediction output modules, and the pre-trained model to obtain an initial image classification model; A model training module, configured to train the initial image classification model based on the first local image sample set to obtain an intermediate image classification model; a parameter uploading module, configured to upload the local model parameters of the intermediate image classification model to a server, so that the server integrates the local model parameters to obtain global model parameters; a third acquisition module, configured to acquire the global model parameters from the server and update the intermediate image classification model according to the global model parameters; a model distillation module, configured to perform distillation processing on the initial image classification model using the updated intermediate image classification model as a teacher model to obtain a target image classification model; A fourth acquisition module is used to acquire an image to be classified; The image classification module is used to input the image to be classified into the target image classification model to obtain a classification prediction result corresponding to the image to be classified through the target image classification model.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program, which is executed by the at least one processor to enable the at least one processor to perform the image classification method for incremental learning according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image classification method for incremental learning according to any one of claims 1 to 7 is implemented.

Images