Classification device, method and related products for classifying images based on meta-learning

Abstract

The present disclosure provides a classification device, method and related products for classifying images based on meta-learning. The classification device includes a shared feature extractor for extracting shared features of an image to be classified, a main classifier for extracting main feature data related to a main classification task from shared feature data and performing the main classification task based on the main feature data to achieve image classification, and an auxiliary classifier for extracting auxiliary feature data related to an auxiliary classification task from shared feature data to perform an auxiliary classification task for the image, wherein the auxiliary classification task is used to perform fine-tuning on the parameters of the shared feature extractor in the meta-learning. Using the classification scheme of the present disclosure, rapid domain adaptation of the shared feature extractor and the main classifier to images from different domains can be achieved, thereby significantly improving the accuracy and efficiency of the classification scheme of the present disclosure.

Description

Technical Field

[0001] The present disclosure generally relates to the field of image processing. More specifically, the present disclosure relates to a classification device, a classification method, a device and a computer-readable storage medium for classifying images. Background Art

[0002] In recent years, machine learning technology has developed rapidly and has achieved good application performance in some fields. In particular, machine learning technology represented by deep learning has achieved good results in the field of image classification (such as fundus disease classification based on fundus retinal images). However, there are still technical problems that need to be solved in actual classification applications.

[0003] For example, the photographing equipment used to acquire images (such as fundus cameras used to capture fundus retinal images) has different manufacturing processes and technologies due to the non-complete unification of manufacturing standards, resulting in significant differences in the images acquired by photographing equipment of different brands. When the aforementioned photographing equipment is a fundus camera, there is a huge difference in the distribution of fundus images of different brands. For the purpose of explanation, Figure 1 Photos of fundus cameras from different brands are shown. Figure 1 As can be seen from the pictures shown, the background colors of fundus images taken by different fundus cameras are quite different. In actual classification operations, the performance of the classification model for fundus images taken by fundus cameras that are rarely seen or never seen during training is often much worse than those taken by fundus cameras that are common during training. Therefore, when such a classification model is applied to the classification of images taken by unknown or rare fundus cameras, it is difficult to achieve the expected classification effect.

[0004] In view of this, a solution capable of efficiently classifying images is needed in the prior art. Summary of the invention

[0005] At least to solve the technical problems described in the above background technology, the present disclosure proposes a solution for efficiently classifying images. Specifically, by utilizing the classification model of the present disclosure, images (such as fundus images) can be accurately and quickly classified. Furthermore, due to the introduction of meta-learning, the classification scheme of the present disclosure is more easily adapted to the accurate classification of images from different domains (such as from different cameras). In view of this, the present disclosure provides the technical solution of the present disclosure in the following multiple aspects and embodiments.

[0006] In a first aspect, the present disclosure provides a classification device for classifying images based on meta-learning, comprising: a shared feature extractor, which is used to extract shared features of the image to be classified to form shared feature data; a main classifier, which is connected to the shared feature extractor and configured to: receive the shared feature data; extract main feature data related to a main classification task from the shared feature data; and perform the main classification task based on the main feature data to achieve classification of the image; an auxiliary classifier, which is connected to the shared feature extractor and is used to: receive the shared feature data; extract auxiliary feature data related to an auxiliary classification task from the shared feature data; and perform an auxiliary classification task for the image based on the auxiliary feature data, wherein the auxiliary classification task is used to fine-tune the parameters of the shared feature extractor in the meta-learning.

[0007] In one embodiment, the shared feature extractor, the main classifier and the auxiliary classifier are integrally obtained through a training process based on meta-learning.

[0008] In one embodiment, the image is associated with a specific domain from which the image is obtained, and for at least one of the specific domains, the auxiliary classifier is used to perform an auxiliary classification task in a testing phase of the meta-learning so as to perform fine-tuning on the parameters of the shared feature extractor for adaptation to at least one of the specific domains.

[0009] In one embodiment, a total loss function is used to perform parameter updates during the training process, and the total loss function of the classification device is equal to the sum of a main loss function associated with the main classification task and an auxiliary loss function associated with the auxiliary classification task.

[0010] In one embodiment, during the testing phase of the meta-learning, an auxiliary loss function associated with the auxiliary classification task is used to perform the fine-tuning on the parameters of the shared feature extractor.

[0011] In one embodiment, the specific domain is associated with the model, shooting angle, shooting time or shooting light of the image acquisition device used to obtain the image, and different models, shooting angles, shooting times and shooting lights constitute different specific domains.

[0012] In one embodiment, images and image labels obtained under different specific domains form data sets of training domain, validation domain and test domain for the training process, and the shared feature extractor, main classifier and auxiliary classifier use the data sets in the training domain, validation domain and test domain to perform training, validation and testing operations accordingly.

[0013] In one embodiment, the auxiliary classifier is used to perform angle classification for a rotation angle of the image.

[0014] In one embodiment, the image is a fundus image of the fundus.

[0015] In a second aspect, the present disclosure provides a classification method for performing image classification using a meta-learning-based classification device, wherein the device includes a shared feature extractor and a main classifier and an auxiliary classifier respectively connected to the shared feature extractor, and the method includes: using the shared feature extractor to extract shared features of the image to be classified to form shared feature data; and using the main classifier and the auxiliary classifier to respectively perform a main classification task and an auxiliary classification task, wherein in the execution of the main classification task, the main classifier is used to perform: receiving the shared feature data; extracting main feature data related to the main classification task from the shared feature data; and performing the main classification task based on the main feature data to achieve classification of the image, wherein in the execution of the auxiliary classification task, the auxiliary classifier is used to perform: receiving the shared feature data; extracting auxiliary feature data related to the auxiliary classification task from the shared feature data; and performing an auxiliary classification task for the image based on the auxiliary feature data, wherein in the meta-learning, the method also includes using the auxiliary classification task to fine-tune the parameters of the shared feature extractor.

[0016] In one embodiment, the classification method further comprises using a meta-learning-based training process to train the shared feature extractor, the main classifier and the auxiliary classifier as a whole.

[0017] In one embodiment, the image is associated with a specific domain from which the image is obtained, and before the main classifier performs the main classification task for the image of the specific domain, the classification method further comprises:

[0018] For at least one of the specific domains, the auxiliary classifier is used to perform an auxiliary classification task during a test phase of the meta-learning to fine-tune parameters of the shared feature extractor so that the shared feature extractor is adapted to the specific domain.

[0019] In one embodiment, the method further includes performing parameter updates using a total loss function of the classification device during a meta-learning training process, wherein the total loss function is equal to the sum of a main loss function associated with the main classification task and an auxiliary loss function associated with the auxiliary classification task.

[0020] In one embodiment, the method further comprises, during a testing phase of the meta-learning, performing the fine-tuning on parameters of the shared feature extractor using an auxiliary loss function associated with the auxiliary classification task.

[0021] In one embodiment, the specific domain is associated with the model, shooting angle, shooting time or shooting light of the image acquisition device used to obtain the image, and different models, shooting angles, shooting times and shooting lights constitute different specific domains.

[0022] In one embodiment, images and image labels obtained under different specific domains form data sets of training domain, validation domain and test domain for the training process, and the method includes: enabling the shared feature extractor, main classifier and auxiliary classifier to use the data sets in the training domain, validation domain and test domain to perform training, validation and testing operations accordingly.

[0023] In one embodiment, wherein the auxiliary classification task includes angle classification, the method further comprises using the auxiliary classifier to perform angle classification for a rotation angle of the image.

[0024] In one embodiment, the image is a fundus image of the fundus.

[0025] In a third aspect, the present disclosure provides a device for performing image classification, comprising: a processor; a memory for storing computer-executable instructions, which, when executed by the processor, implements the classification method described in the second aspect and its multiple embodiments.

[0026] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing program instructions for classifying images. When the program instructions are executed by a processor, the classification method according to the third aspect and its multiple embodiments is implemented.

[0027] By utilizing the schemes described in the above-mentioned aspects of the present disclosure and in its embodiments, efficient and accurate classification of images can be achieved. Specifically, the present disclosure can achieve efficient and rapid classification of images from different domains by utilizing the coordinated use of the main classifier and the auxiliary classifier. Furthermore, since the classification device is trained by meta-learning, the learning of the device for classification obtained by the present disclosure is more effective in the training phase, and can quickly adapt to and perform classification operations of various tasks. In addition, the classification device of the present disclosure does not have too many restrictions in terms of network structure, and when the classification device of the present disclosure is constructed based on a convolutional neural network, effective extraction of image features can be achieved, thereby facilitating subsequent accurate classification. In one embodiment, when the scheme of the present disclosure is applied to the classification of fundus images, accurate classification of fundus images from different domains can be achieved, so that the eye diseases or symptoms corresponding to the fundus images can be effectively determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0029] Figure 1 is a schematic diagram showing a fundus picture taken using a fundus camera;

[0030] Figure 2 is a schematic block diagram showing a classification device for classifying images based on meta-learning according to an embodiment of the present disclosure;

[0031] Figure 3 is a schematic block diagram showing a training phase of a classification device according to an embodiment of the present disclosure;

[0032] Figure 4 is a schematic network structure block diagram showing a classification device according to an embodiment of the present disclosure;

[0033] Figure 5 is a flow chart showing a classification method for classifying an image according to an embodiment of the present disclosure;

[0034] Figure 6 is a block diagram showing a device for classifying an image according to an embodiment of the present disclosure; and

[0035] Figure 7 is a block diagram illustrating a system for classifying an image according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments are now described with reference to the accompanying drawings. It should be understood that for simplicity and clarity of explanation, reference numerals may be repeated in the accompanying drawings to indicate corresponding or similar elements where deemed appropriate. In addition, the present application sets forth many specific details in order to provide a thorough understanding of the embodiments described herein. However, it will be appreciated by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other cases, well-known methods, processes, and components are not described in detail to avoid blurring the embodiments described herein. Moreover, the description should not be construed as limiting the scope of the embodiments described herein.

[0037] As mentioned above, in order to achieve efficient and accurate classification of images, the solution disclosed in the present invention proposes a new classification model based on a shared feature extractor, a main classifier and an auxiliary classifier, wherein the shared feature extractor extracts shared features, and the main classifier and the auxiliary classifier respectively perform the main classification task and the auxiliary classification task, thereby achieving accurate classification of images, such as associating images with corresponding labels.

[0038] In one or more embodiments, by utilizing a neural network architecture in machine learning (e.g., a convolutional neural network) to construct a classification model and with the aid of a meta-learning training mechanism, the classification scheme of the present disclosure can be fine-tuned to quickly adapt to images from different domains. In the context of the present disclosure, the statistical distributions of the aforementioned images representing different domains are different. For example, images obtained from different camera brands, different shooting angles of the same camera (such as a normal angle, a 45-degree angle, or a wide angle), different shooting time points (or shooting in the evening and shooting during the day), or different shooting lights can all be considered to be images from different domains. Due to the ability to quickly adapt to different domains, the classification scheme of the present disclosure has better applicability and scalability than the prior art, and can thus be applied to image classification processing tasks in various fields.

[0039] In an image classification application scenario, the scheme of the present disclosure can be used to classify retinal fundus images (referred to as "fundus images"), for example, classifying fundus images containing different fundus diseases by disease type. These diseases, for example, involve macular degeneration, retinal tumors, diabetic retinopathy, arteriosclerosis, etc. As is known in the art, fundus images are an important imaging means for observing the retina and diagnosing fundus diseases. In view of this, the present disclosure classifies fundus images through classification devices and equipment, thereby realizing rapid detection and accurate identification of fundus diseases.

[0040] The scheme of the present disclosure and its multiple embodiments will be described in detail below with reference to the accompanying drawings.

[0041] Figure 2 is a schematic block diagram showing a classification device 200 for classifying an image according to an embodiment of the present disclosure.

[0042] like Figure 2 As shown, the classification device 200 of the present disclosure may include a shared feature extractor 202, a main classifier 204, and an auxiliary classifier 206. In one embodiment, one or more of the shared feature extractor, the main classifier, and the auxiliary classifier may be implemented by a neural network architecture in machine learning. For example, the neural network architecture may be a convolutional neural network model for image feature extraction and classification, and it includes various layers and operators for extracting image features, such as convolutional layers and convolutional operators and pooling layers and pooling operators. As mentioned above, the scheme of the present disclosure does not impose specific restrictions on the structure of a specific convolutional neural network model.

[0043] In one embodiment, the aforementioned shared feature extractor is used to extract shared features of the image to be classified to form shared feature data. Here, the shared feature data may be image data containing low resolution or image data containing low-level features. In an application scenario, the shared feature extractor here may be implemented as a convolutional neural network comprising one or more (e.g., two) cascaded convolutional layers and / or pooling layers to extract shared feature data from an image (e.g., a fundus photograph).

[0044] In one embodiment, the main classifier connected to the above-mentioned shared feature extractor can be configured to receive shared feature data (such as the low-resolution data or low-level feature data above). Then, the main classifier extracts the main feature data related to the main classification task from the shared feature data. Here, the main feature data can be high-resolution or high-level feature data related to a specific classification task. Then, the main classifier can perform the main classification task based on the main feature data to achieve classification of the image.

[0045] In one implementation scenario, when the aforementioned image is a fundus image, the main feature data here may be feature data related to determining a specific fundus disease, such as feature data related to a macular or retinal tumor. Based on this, the main classifier can associate the fundus image with a specific disease or condition to complete the classification of the image, that is, the identification of the fundus disease. In one implementation scenario, the main classifier here can also be constructed as a convolutional neural network, and includes cascaded convolutional layers, pooling layers and / or fully connected layers, so as to achieve the extraction of the main features and the final classification of the image.

[0046] In one embodiment, the auxiliary classifier connected to the above-mentioned shared feature extractor can be used to receive shared feature data and extract auxiliary feature data related to the auxiliary classification task from the shared feature data. Then, the auxiliary classifier can perform the auxiliary classification task for the image based on the auxiliary feature data. In an application scenario, before the main classifier performs the main classification task for the image of a specific domain, the auxiliary classifier of the present disclosure performs the auxiliary classification task in the test phase of meta-learning, so as to fine-tune the parameters of the shared feature extractor for the specific domain of the image, so that the updated parameters of the shared feature extractor are adapted to the specific domain, thereby promoting the main classifier to adapt to the image classification task of the specific domain, thereby improving the classification accuracy and efficiency of the main classifier. In one embodiment, the aforementioned parameter fine-tuning can be achieved by minimizing the auxiliary loss function associated with the auxiliary classification task. It can be understood that when the auxiliary classification task is performed for multiple specific domains in the test phase to adjust the parameters of the shared feature extractor, the resulting shared feature extractor or main classifier will quickly adapt to the image classification of the aforementioned multiple specific domains, thereby expanding the application scenario and scalability of the classification device of the present disclosure.

[0047] In one implementation scenario, in order to promote the above-mentioned main classifier to be quickly applicable to the classification task of images from a specific domain, the auxiliary classifier of the present disclosure can perform a prediction task for automatically generating labels for images, such as a classification task of the rotation angle of the image. In this case, the auxiliary classifier essentially performs a self-supervised task without additional annotation. As for the construction of the auxiliary classifier, similar to the construction of the shared feature extractor and the main classifier, it can also be implemented by a convolutional neural network, and for example includes a cascaded convolution layer, a pooling layer, and a fully connected layer. By performing such a classification task of the rotation angle and minimizing the aforementioned auxiliary loss function, the model parameters of the shared feature extractor can be fine-tuned to quickly adapt to the specific domain of the image to be classified. Thus, when the main classifier performs the main classification task on image classification, it can also quickly adapt to the specific domain of the image, thereby enhancing the efficiency of classification and improving the accuracy of the classification results.

[0048] In order to achieve efficient classification of images, in an implementation scenario, the present disclosure proposes combining the above Figure 2 The classification device described is trained, in particular, training based on meta-learning, so that the classification device of the present disclosure can quickly adapt to images from different domains. Regarding the meta-learning ("meta learning") used in the present disclosure, its core is to let machines (such as various neural network models) learn to learn. For example, in a meta-learning scenario, the machine will acquire a certain learning ability after learning 100 tasks, so that when learning the 101st task, it will learn the 101st task faster or better based on the learning ability acquired when learning the first 100 tasks. When meta-learning is applied to the scheme of the present disclosure, learning can be performed on multiple domains based on the same network architecture (such as a convolutional neural network), so that an efficient and fast classification device can be obtained.

[0049] In order to better understand the solution of the present disclosure, Figure 3 The training process of the classification device of the present disclosure based on meta-learning is described by taking the camera model of the domain as an example. In this training scenario, the classification device of the present disclosure can be regarded as Figure 3 The multi-task classification model shown in . Therefore, the above combined Figure 2 The same description of the classification device applies to Figure 3 The multi-task learning network in , and therefore the same content is not repeated here.

[0050] like Figure 3As shown in , in order to achieve an efficient training process so as to obtain the best multi-task classification network model based on meta-learning, the present disclosure proposes to divide the meta-learning dataset into domains (i.e., camera models in this example, such as cameras T-1, T-2, ..., TN, V-1, V-2, ..., VM, Te-1, Te-2, ..., Te-Q) as basic units, thereby obtaining Figure 3 The training domain, validation domain, and test domain shown in .

[0051] Specifically, for each training domain, a batch of images from the domain (i.e., images in the context of this application, such as fundus images) can be collected, and the batch of images can be divided into a support set ("support set") and a query set ("query set"), which are used to update classifier parameters and test classifiers in the training operation, respectively. Similarly, images from the validation domain and the test domain can be collected as the validation set and test set of the multi-task classification network model for verification and testing of the model during the training phase.

[0052] In order to further understand the establishment of the data set in the training phase of the multi-task classification network model of the present invention, fundus images from different domains are shown in the form of a table below. As described above, the domains shown in the table also include training domains, verification domains, and test domains. Furthermore, the aforementioned domains contain data sets consisting of fundus images from their respective domains. In this example, the aforementioned data set consists of fundus images (or "fundus retinal images") and fundus disease labels associated with the images.

[0053]

[0054]

[0055] Combination of the above Figure 3 The training data set for establishing the classification device based on meta-learning of the present disclosure is described below. Figure 4 The network architecture block diagram shown is used to describe the training process of the classification device of the disclosed solution, which involves adjusting the parameters of the shared feature extractor, the main classifier and the auxiliary classifier in a manner to minimize the loss function.

[0056] like Figure 4As shown, a shared feature extractor 202, a main classifier 204, and an auxiliary classifier 206 are respectively shown in the form of an exemplary neural network architecture. As mentioned above, in one embodiment, the shared feature extractor (or shared feature extraction layer), the main classifier, and the auxiliary classifier of the present disclosure can be constructed in the form of a convolutional neural network, and include, for example, a convolutional layer, a pooling layer, and / or a fully connected layer commonly used in a convolutional neural network, and each layer can include a different number of neuron nodes as shown in the figure. Thus, the extraction of features of different levels of the image can be achieved, such as the extraction of shared features, main features, and auxiliary features of the present disclosure.

[0057] In order to achieve parameter adjustment in the meta-learning process, the present disclosure proposes to represent the training data as {(x1,y1),...,(x n ,y n )}, where x represents the image and y represents the corresponding label. Next, the network model including the shared feature extractor, the main classifier and the auxiliary classifier is represented by the function f(Θ), where the network parameters are represented as Θ = {θe, θm, θa}, representing the parameters of the k-layer shared feature extractor θe = (θ1, ..., θ k ), the parameter of the main classifier of layer h is θm = (θ k+1 ,...,θ k+h ), and the parameters of the auxiliary classifier of layer r θa=(θ′ k+1 ,...,θ′ k+r ).

[0058] For each training domain, the training of the multi-task classification network model of the present disclosure can be carried out in a standard multi-task learning training mode, and can be trained on the main task and the auxiliary task simultaneously, so that the total loss function of the multi-task model of the present disclosure can be equal to the sum of the main loss function associated with the main task and the auxiliary loss function associated with the auxiliary task. Based on this, the training problem of the present disclosure can be expressed as shown in the following formula (1):

[0059]

[0060] Where n represents the number of samples in one training (“batch size”), and L(f(θ)) represents the total loss function. As can be seen from formula (1), the total loss function consists of two parts, where l m (x i ,y i ; θe,θm) represents the main loss function of the main task, and l a (x i ; θe,θa) represents the auxiliary loss function of the auxiliary task.

[0061] In an implementation scenario, as mentioned above, the main task of the present disclosure may be a task of identifying fundus diseases, while the auxiliary task is a task used to assist the main task in quickly adapting to the domain. In an application, in order to save manpower and computational costs, the present disclosure proposes to use the task of automatically generating labels as an auxiliary classification task, such as a rotation angle prediction task. Specifically, the rotation angle prediction task refers to manually rotating an image by a certain degree, and then using a model to predict the rotation degree. Preferably, when the classified image is a fundus image, considering the particularity of the fundus image, the present disclosure proposes to limit the rotation degree to 0 degrees and 90 degrees. Therefore, the rotation angle prediction task requires the fundus image that is randomly rotated by 0 degrees or 90 degrees to classify the rotation angle. Since the rotation angle is known, the auxiliary task in this scenario can be regarded as a self-supervised task and does not require additional labeling.

[0062] After building the combination Figure 3 and Figure 4 After describing the data set and the multi-task classification network model respectively, the present disclosure proposes to train the multi-task classification network model by meta-learning training, so that the classification network can quickly adapt to the domain of the test data (i.e., the test image) after fine-tuning during testing, thereby obtaining a more accurate prediction result. In one embodiment, the present disclosure uses meta-learning training to allow the classification network model to quickly adapt to a new domain through one or a few steps of gradient descent training, i.e., the loss on the new domain is minimized.

[0063] For exemplary purposes only, the following is a pseudo code showing the process of updating network parameters in the training of the present disclosure.

[0064]

[0065] It can be seen from the above pseudo code that the multi-task classification network model disclosed in the present invention implements the update of network model parameters through, for example, a gradient descent algorithm.

[0066] After executing the above training process, the classification device of the present invention, that is, the trained multi-task classification network model, can be obtained. Significantly different from the existing deep learning classification model testing, the testing of the multi-task classification network model based on meta-learning training of the present invention can be divided into two steps, namely, the training in the test phase ("test-time training") and model inference ("inference").

[0067] Specifically, the training in the test phase is to fine-tune the parameters of the multi-task classification network model to make it closer to the domain where the test image is located. In simple terms, the training in the test phase is to fine-tune the parameters of the shared feature extractor by minimizing the loss of the auxiliary task (such as the self-supervised rotation angle prediction task as described above) in the multi-task classification network model of the present disclosure on the test image, as shown in the following formula (2):

[0068]

[0069] Usually in this fine-tuning step, only one or a few steps of the gradient descent algorithm need to be executed to complete the optimization of the multi-task classification network model of the present invention. After that, the test image can be input into the fine-tuned multi-task classification network model to obtain the output of the main classifier, that is, the classification result (or prediction result) of the test image. Furthermore, when the multi-task classification network model of the present invention is applied to the classification and prediction of fundus diseases, the classification device of the present invention can be regarded as a fundus disease classifier based on a convolutional neural network. The output of the main classifier of the fundus disease classifier is the prediction result of the fundus disease classification of the fundus image.

[0070] Figure 5 1 is a flow chart showing a classification method 500 for classifying images according to an embodiment of the present disclosure. It is understood that the classification method 500 here can be executed by a general-purpose processor or a special-purpose processor (such as an artificial intelligence processor) to classify the image. It is further understood that the classification method 500 involves using the present disclosure in combination with Figure 2-Figure 4 The classification is carried out by using the classification device described, so the above description of the classification device of the present invention is also applicable to the description of the classification method below.

[0071] like Figure 5 As shown, at step S502, the shared feature extractor of the classification device is used to extract the shared features of the image to be classified to form shared feature data. Then, at step S504, the main classifier and the auxiliary classifier of the classification device are used to perform the main classification task and the auxiliary classification task respectively. As described above, in the execution of the main classification task, the main classifier is made to receive the shared feature data at step S506, and then extract the main feature data related to the main classification task from the shared feature data at step S508. Finally, at step S510, the main classifier is made to perform the main classification task based on the main feature data to achieve the classification of the image. In the execution of the auxiliary classification task, the auxiliary classifier is made to receive the shared feature data at step S512, and extract the auxiliary feature data related to the auxiliary classification task from the shared feature data at step S514, and perform the auxiliary classification task for the image based on the auxiliary feature data at S516.

[0072] Although not shown in the figure, in one implementation scenario, in meta-learning, the method 500 also includes using an auxiliary classification task to fine-tune the parameters of the shared feature extractor. Specifically, before the main classifier performs the main classification task for the image of the specific domain, the classification method can use the auxiliary classifier to perform an auxiliary classification task for at least one of the specific domains during the test phase of the meta-learning to fine-tune the parameters of the shared feature extractor so that the shared feature extractor is adapted to the specific domain. Similar to the above description, in one implementation, the auxiliary loss function associated with the auxiliary classification task can be used during the test phase of the meta-learning to perform the fine-tuning on the parameters of the shared feature extractor. The auxiliary classification task can be, for example, angle classification without manual annotation.

[0073] Figure 6 6 is a block diagram showing a device 600 for classifying images according to an embodiment of the present disclosure. To facilitate understanding of the scheme of the present disclosure, as an example, the figure also shows a fundus camera 606 for collecting fundus images, which is used to transmit image data about the fundus image to the device 600 for classification of the present disclosure so as to output a classification result.

[0074] like Figure 6 As shown in FIG. 6 , the device 600 includes a processor 601 and a memory 602, wherein the memory 602 stores the computer-coded shared feature extractor 202, the main classifier 204, and the auxiliary classifier 206. When the processor executes the computer code stored in the memory, the above-mentioned Figure 5 For example, in the meta-learning training process, the device 600 may receive training set, validation set, and test set data (eg, Figure 3 As shown in the figure), in order to instruct the coded classification device to perform training, and to utilize the auxiliary task classification to fine-tune the parameters of the shared feature extractor (such as the aforementioned θe) during training in the test phase to achieve rapid adaptation to different domains.

[0075] Figure 7 is a block diagram showing a system 700 for classifying images according to an embodiment of the present disclosure. Figure 7 As shown in FIG. 7 , the system 700 may include a device 701 for classification according to an embodiment of the present disclosure, as well as its peripheral devices and an external network. It can be understood that the device 701 here is the above-mentioned Figure 6 A specific embodiment of the device 600 is described, so the description of the device 600 is also applicable to Figure 7 The device 701 shown in FIG.

[0076] like Figure 7 As shown in , the device 701 may include a CPU 7011, which may be a general-purpose CPU, a dedicated CPU, or other information processing and program execution units. Further, the device 701 may also include a large-capacity memory 7012 and a read-only memory ROM 7013, wherein the large-capacity memory 7012 may be configured to store various types of data, including the data of the training set, the validation set, and the test set of the present disclosure and various programs required for running the classification device of the present disclosure, and the ROM 7013 may be configured to store the power-on self-test of the device 701, the initialization of each functional module in the system, the basic input / output driver of the system, and the data required for booting the operating system.

[0077] Further, the device 701 also includes other hardware platforms or components, such as the tensor processing unit ("TPU") 7014, the graphics processing unit ("GPU") 7015, the field programmable gate array ("FPGA") 7016, and the machine learning unit ("MLU") 7017 shown. It is understood that although a variety of hardware platforms or components are shown in the device 701, they are only exemplary and not restrictive, and those skilled in the art can add or remove corresponding hardware according to actual needs. For example, the device 701 may only include a CPU as a hardware platform to execute the classification scheme of the present disclosure.

[0078] The device 701 of the present disclosure may also include a communication interface 7018, so that it can be connected to a local area network / wireless local area network (LAN / WLAN) 705 through the communication interface 7018, and then connected to a local server 706 or to the Internet ("Internet") 707 through the LAN / WLAN. Alternatively or additionally, the device 701 of the present disclosure may also be directly connected to the Internet or a cellular network through the communication interface 7018 based on wireless communication technology, such as wireless communication technology based on the 3rd generation ("3G"), 4th generation ("4G") or 5th generation ("5G") generation. In some application scenarios, the device 701 of the present disclosure may also access a server 708 of an external network and a possible database 709 as needed to obtain various known neural network models, data and modules, and may remotely store various data, including the training data and classification result data involved in the present disclosure.

[0079] In some application scenarios, the peripheral devices of the device 701 of the present disclosure may include a display device 702, an input device 703, and a data transmission interface 704. In one embodiment, the display device 702 may include, for example, one or more speakers and / or one or more visual displays, which are configured to provide voice prompts and / or image video displays for the computing process or final classification results of the device of the present disclosure. The input device 703 may include, for example, a keyboard, a mouse, a microphone, a gesture capture camera, a fundus camera, or other input buttons or controls, which are configured to receive input or user instructions for image data. The data transmission interface 704 may include, for example, a serial interface, a parallel interface or a universal serial bus interface ("USB"), a small computer system interface ("SCSI"), a serial ATA, a FireWire ("FireWire"), a PCI Express, and a high-definition multimedia interface ("HDMI"), etc., which are configured for data transmission and interaction with other devices or systems. According to the scheme of the present disclosure, the data transmission interface 704 may also receive various types of image data from an image acquisition device, and send the classification results obtained after performing the classification operation to the image acquisition device.

[0080] The CPU 7011, the mass storage 7012, the read-only memory ROM 7013, the TPU 7014, the GPU 7015, the FPGA 7016, the MLU 7017 and the communication interface 7018 of the device 701 of the present disclosure may be interconnected via a bus 7019, and data interaction with peripheral devices may be achieved via the bus. In one embodiment, the CPU 7011 may control other hardware components and peripheral devices in the device 701 via the bus 7019.

[0081] In a working scenario, the processor CPU 7011 of the device 701 of the present disclosure can receive training data or image data to be classified through the input device 703 or the data transmission interface 704, and call the computer program instructions or codes (such as the coded classification device of the present disclosure) stored in the memory 7012 to train or classify the received data to obtain a loss function or classification result for adjusting network parameters. In addition, the device 701 can also upload the coded multi-task classification network model or classification result obtained after training to the network, such as the remote database 709, through the communication interface 7018.

[0082] It should also be understood that any module, unit, component, server, computer, terminal or device that executes instructions of the examples of the present disclosure may include or otherwise access computer-readable media, such as storage media, computer storage media or data storage devices (removable and / or non-removable) such as disks, optical disks or tapes. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules or other data.

[0083] Based on the above, the present disclosure also discloses a computer-readable storage medium in which program instructions are stored, and the program instructions are suitable for being loaded by a processor and executing the present disclosure in combination with Figure 5 The method and its various embodiments are described.

[0084] The computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as RRAM (Resistive Random Access Memory), DRAM (Dynamic Random Access Memory), SRAM (Static Random-Access Memory), EDRAM (Enhanced Dynamic Random Access Memory), HBM (High-Bandwidth Memory), HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by the application, the module, or both. Any such computer storage medium may be part of the device or accessible or connectable to the device. Any application or module described in the present disclosure may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0085] It should be understood that the terms "include" and "comprises" used in the specification and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections. It should also be understood that the terms used in this disclosure specification are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. As used in the disclosure specification and claims of the present disclosure, the singular forms of "one", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise. The term "and / or" used in the disclosure specification and claims of the present disclosure refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0086] Although the embodiments of the present disclosure are as described above, the contents are only examples used to facilitate understanding of the present disclosure, and are not intended to limit the scope and application scenarios of the present disclosure. Any technician in the technical field described in the present disclosure may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be based on the scope defined in the attached claims.

Claims

1. A classification device for classifying images based on meta-learning, comprising: A shared feature extractor, which is used to extract shared features of the image to be classified to form shared feature data; A main classifier connected to the shared feature extractor and configured to: Receiving the shared feature data; Extracting main feature data related to the main classification task from the shared feature data; and Performing a main classification task based on the main feature data to achieve classification of the image; Auxiliary classifier, which is connected to the shared feature extractor and is used to: Receiving the shared feature data; Extracting auxiliary feature data related to the auxiliary classification task from the shared feature data; performing an auxiliary classification task for the image based on the auxiliary feature data, wherein the auxiliary classification task is used to perform fine-tuning on parameters of the shared feature extractor in the meta-learning; The shared feature extractor, the main classifier and the auxiliary classifier are obtained as a whole through a training process based on the meta-learning.

2. The classification device according to claim 1, wherein the image is associated with a specific domain from which the image is obtained, and for at least one of the specific domains, the auxiliary classifier is used to perform an auxiliary classification task in a testing phase of the meta-learning so as to perform fine-tuning on the parameters of the shared feature extractor for adaptation to at least one of the specific domains.

3. The classification device according to claim 2, wherein the total loss function is used to perform parameter updates during the training process, and the total loss function of the classification device is equal to the sum of the main loss function associated with the main classification task and the auxiliary loss function associated with the auxiliary classification task. 4 . The classification apparatus according to claim 3 , wherein in a testing phase of the meta-learning, an auxiliary loss function associated with the auxiliary classification task is used to perform the fine-tuning on the parameters of the shared feature extractor.

5. The classification device according to claim 2, wherein the specific domain is associated with the model, shooting angle, shooting time or shooting light of the image acquisition device used to obtain the image, and different models, shooting angles, shooting times and shooting lights constitute different specific domains.

6. The classification device according to claim 5, wherein the images and image labels obtained under different specific domains form data sets of training domain, verification domain and test domain for the training process, and the shared feature extractor, main classifier and auxiliary classifier use the data sets in the training domain, verification domain and test domain to perform training, verification and testing operations accordingly.

7. The classification device according to any one of claims 1 to 6, wherein the auxiliary classifier is used to perform angle classification for the rotation angle of the image. The classification device according to claim 7 , wherein the image is a fundus image regarding the fundus.

9. A classification method for performing image classification using a classification device based on meta-learning, wherein the classification device comprises a shared feature extractor and a main classifier and an auxiliary classifier respectively connected to the shared feature extractor, the classification method comprising: Using a shared feature extractor to extract shared features of the image to be classified to form shared feature data; as well as Using the main classifier and the auxiliary classifier to perform the main classification task and the auxiliary classification task respectively, In the execution of the main classification task, the main classifier is used to perform: Receiving the shared feature data; Extracting main feature data related to the main classification task from the shared feature data; and performing a main classification task based on the main feature data to classify the image, In the execution of the auxiliary classification task, the auxiliary classifier is used to perform: receiving the shared feature data; extracting auxiliary feature data related to the auxiliary classification task from the shared feature data; and performing an auxiliary classification task for the image based on the auxiliary feature data, Wherein in the meta-learning, the method further comprises using an auxiliary classification task to perform fine-tuning on parameters of the shared feature extractor; The shared feature extractor, the main classifier and the auxiliary classifier are obtained as a whole through a training process based on the meta-learning.

10. The classification method according to claim 9, further comprising: The shared feature extractor, the main classifier and the auxiliary classifier are trained as a whole using a meta-learning based training process.

11. The classification method according to claim 9, wherein the image is associated with a specific domain from which the image is obtained, and before the main classifier performs a main classification task for the image of the specific domain, the classification method further comprises: For at least one of the specific domains, the auxiliary classifier is used to perform an auxiliary classification task during a test phase of the meta-learning to fine-tune parameters of the shared feature extractor so that the shared feature extractor is adapted to the specific domain.

12. The classification method according to claim 11, further comprising: During the meta-learning training process, the total loss function of the classification device is used to perform parameter updating, wherein the total loss function is equal to the sum of the main loss function associated with the main classification task and the auxiliary loss function associated with the auxiliary classification task.

13. The classification method according to claim 12, further comprising: During a testing phase of the meta-learning, the fine-tuning is performed on parameters of the shared feature extractor using an auxiliary loss function associated with the auxiliary classification task.

14. The classification method according to claim 11, wherein the specific domain is associated with the model, shooting angle, shooting time or shooting light of the image acquisition device used to obtain the image, and different models, shooting angles, shooting times and shooting lights constitute different specific domains.

15. The classification method according to claim 14, wherein the images and image labels obtained in different specific domains form data sets of training domains, validation domains and test domains used in the training process, and the classification method comprises: The shared feature extractor, the main classifier and the auxiliary classifier are enabled to perform training, verification and testing operations accordingly using the data sets in the training domain, the verification domain and the test domain.

16. The classification method according to any one of claims 9 to 15, wherein the auxiliary classification task comprises angle classification, and the method further comprises: Angle classification for the rotation angle of the image is performed using the auxiliary classifier. The classification method according to claim 16 , wherein the image is a fundus image of an ocular fundus.

18. An apparatus for performing image classification, comprising: processor; A memory for storing computer-executable instructions, which, when executed by the processor, implements the classification method according to any one of claims 9 to 17.

19. A computer-readable storage medium storing program instructions for classifying images, wherein when the program instructions are executed by a processor, the classification method according to any one of claims 9 to 17 is implemented.

Images