Eye image classification method based on self-supervised learning and related equipment

Abstract

The invention provides an eye image classification method based on self-supervised learning and related equipment, and the method comprises the steps: carrying out the block mixing of a group of unlabeled sample images, training an upstream task model, namely a supervision task model, through the obtained feature map of a group of mixed images, recovering the feature map of an original sample image, namely a de-mixing feature map, and carrying out the recognition of an eye image. The monitoring information mining efficiency and accuracy are improved, then the monitoring task model is transferred and learned into a pre-training model of a downstream task, namely, an eye image classification task, and a corresponding eye image classification model is obtained after model parameter fine adjustment, so that a large number of sample images do not need to be labeled, and the manual labeling cost is greatly saved; and the model learning efficiency is improved. Therefore, the to-be-recognized eye image is input into the eye image classification model, the category prediction information can be quickly and accurately obtained, and the classification result of the to-be-recognized eye image is output accordingly.

Description

Technical Field

[0001] The present invention mainly relates to the field of image processing, and in particular to an eye image classification method based on self-supervised learning and related equipment. Background Art

[0002] Ocular surface disease (OSD) refers to diseases that damage the normal structure and function of the cornea, conjunctiva and other ocular surfaces. It is usually diagnosed using slit lamps and medical equipment based on optical coherence tomography to capture images of the patient's eyes. Ophthalmologists then observe the eye images to see if there are any pathological features to determine which ocular surface disease the patient has. This requires a high level of professional skills from ophthalmologists and cannot guarantee the reliability and accuracy of the judgment results.

[0003] In this regard, it is proposed to use a large amount of manually annotated data with category labels to obtain a classification model for identifying the category information of the input image through supervised learning. In this way, the collected eye images can be input into the classification model to obtain the category information of the eye image, such as whether it belongs to a certain ocular surface disease category, to assist ophthalmologists in quickly and accurately determining the patient's condition. However, since the acquisition process of manually annotated data is time-consuming and labor-intensive, and the amount of data obtained is limited, the learning efficiency and reliability of the classification model are affected. Summary of the invention

[0004] In order to solve the above problems, this application proposes the following technical solutions:

[0005] On the one hand, the present application proposes an eye image classification method based on self-supervised learning, the method comprising:

[0006] Obtaining an eye image to be identified;

[0007] The eye image to be identified is processed by an eye image classification model obtained based on self-supervised learning to obtain category prediction information; wherein the eye image classification model is obtained by transferring the pre-trained supervised task model to the eye image classification task model and fine-tuning the model parameters using sample images, and the supervised task model is obtained by supervised learning of the predicted positions of the image blocks of a group of mixed images, and the group of mixed images is obtained by mixing a group of sample images in blocks;

[0008] The category prediction information is used to output the category result of the eye image to be identified.

[0009] Optionally, the step of mixing a group of sample images into blocks to obtain a group of mixed images includes:

[0010] Obtain a set of sample images; the set of sample images includes a plurality of sample images;

[0011] Dividing each of the sample images in the set of sample images into the same number of image blocks;

[0012] Based on the number of image blocks obtained by segmenting each of the sample images, an image block mixing mask for the group of sample images is obtained; the image block mixing mask is composed of image identifiers of the plurality of sample images;

[0013] Based on the image block mixing mask, the image blocks in the set of sample images are mixed to obtain a corresponding set of mixed images.

[0014] Optionally, obtaining the image block mixing mask for the group of sample images based on the number of image blocks obtained by segmenting each of the sample images includes:

[0015] Determine an image identifier corresponding to each sample image in the set of sample images;

[0016] Based on the number of image blocks obtained by segmenting the sample image, a sub-mixing mask for each sample image is randomly generated; the number of image identifiers included in the sub-mixing mask is the same as the number of image blocks into which one sample image is segmented;

[0017] An image block blending mask for the set of sample images is constructed by the sub-blending masks.

[0018] Optionally, performing mixing processing on the image blocks in the set of sample images based on the image block mixing mask to obtain a corresponding set of mixed images includes:

[0019] According to the image identifiers at different positions in each of the sub-mixed masks, a mixed image is formed by the image blocks at corresponding positions after the sample image with the corresponding image identifier is segmented.

[0020] Optionally, the supervised task model is obtained by performing supervised learning on the predicted positions of the image blocks of a group of mixed images, including:

[0021] Inputting the set of mixed images into a coding network for feature extraction to obtain a corresponding feature map;

[0022] Based on the image block mixing mask, obtaining an image block restoration mask for the set of mixed images;

[0023] The feature map and the image block restoration mask are input into a decoding network, and supervised learning is performed on the image identifier of the sample image to which the predicted corresponding image block belongs, so as to obtain an auxiliary task model.

[0024] Optionally, the step of inputting the feature map and the image block restoration mask into a decoding network, and obtaining an auxiliary task model by performing supervised learning on an image identifier of the sample image to which the predicted corresponding image block belongs, comprises:

[0025] Performing semantic analysis on each feature block included in the feature map based on a decoding network to identify a predicted image identifier of the sample image to which the corresponding image block belongs;

[0026] Based on a cross entropy loss function, the predicted image identifier and the image identifier of the image block corresponding to the image block restoration mask are calculated to obtain a position prediction loss corresponding to the image block;

[0027] By minimizing the position prediction loss, the decoding network is trained to obtain an auxiliary task model for the eye image classification task.

[0028] Optionally, after transferring the pre-trained supervised task model to the eye image classification task model, fine-tuning the model parameters using sample images to obtain the eye image classification model includes:

[0029] Transfer learning the auxiliary task model to a pre-trained model of a downstream eye image classification task model; the output layer of the downstream eye image classification task model is a classifier;

[0030] The sample images are used to fine-tune the model parameters of the pre-trained model to obtain an eye image classification model.

[0031] Optionally, the classifier is a nonlinear classifier consisting of three fully connected layers;

[0032] The encoding network consists of three convolutional neural network (CNN) layers;

[0033] The decoding network is a convolutional neural network ConvNet.

[0034] On the other hand, the present application also proposes an eye image classification device based on self-supervised learning, the device comprising:

[0035] The module for obtaining the eye image to be identified is used to obtain the eye image to be identified;

[0036] A category prediction information obtaining module, used to process the eye image to be identified through an eye image classification model obtained based on self-supervised learning to obtain category prediction information; wherein the eye image classification model is obtained by transferring the pre-trained supervision task model to the eye image classification task model and fine-tuning the model parameters using sample images, and the supervision task model is obtained by supervised learning of the predicted positions of the image blocks of a group of mixed images, and the group of mixed images is obtained by mixing a group of sample images in blocks;

[0037] The category result output module is used to output the category result of the eye image to be identified by using the category prediction information.

[0038] On the other hand, the present application also proposes an electronic device, the electronic device comprising:

[0039] A communication connection port, used for receiving an eye image to be identified;

[0040] The processor is used to load and execute the computer to implement the following steps:

[0041] Obtaining an eye image to be identified;

[0042] The eye image to be identified is processed by an eye image classification model obtained based on self-supervised learning to obtain category prediction information; wherein the eye image classification model is obtained by transferring the pre-trained supervised task model to the eye image classification task model and fine-tuning the model parameters using sample images, and the supervised task model is obtained by supervised learning of the predicted positions of the image blocks of a group of mixed images, and the group of mixed images is obtained by mixing a group of sample images in blocks;

[0043] The category prediction information is used to output the category result of the eye image to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A flowchart of an optional embodiment 1 of the eye image classification method based on self-supervised learning proposed in this application;

[0046] Figure 2 This is a flow chart of optional embodiment 2 of the eye image classification method based on self-supervised learning proposed in this application;

[0047] Figure 3 An optional schematic diagram of a process for obtaining a mixed image feature map in the eye image classification method based on self-supervised learning proposed in the present application;

[0048] Figure 4 A flowchart of an optional example of a self-supervised learning process in the eye image classification method based on self-supervised learning proposed in the present application;

[0049] Figure 5 This is a flow chart of optional embodiment 3 of the eye image classification method based on self-supervised learning proposed in this application;

[0050] Figure 6 A schematic structural diagram of an optional embodiment of the eye image classification device based on self-supervised learning proposed in this application;

[0051] Figure 7 This is a schematic structural diagram of another optional embodiment of the eye image classification device based on self-supervised learning proposed in this application;

[0052] Figure 8 The present invention is a schematic diagram of the hardware structure of an optional example of an electronic device suitable for the eye image classification method based on self-supervised learning proposed in the present application. DETAILED DESCRIPTION

[0053] In response to the technical problems described in the background technology part, in order to improve the learning efficiency and reliability of the classification model, it is proposed to adopt a self-supervised learning method. By designing auxiliary tasks (Pretext Task, also called upstream tasks), the self-representation features are mined from large-scale unlabeled data as supervision information. The classification network is trained through this constructed supervision information to learn the representation features that are valuable for downstream tasks, thereby improving the feature extraction ability of the classification model and further improving the classification efficiency and reliability.

[0054] Among them, in the process of obtaining supervision information through designed auxiliary tasks, that is, in the pre-training stage of self-supervised representation learning, the present application proposes to slice a group of multiple images without labeled category labels, and then mix the multiple image blocks obtained by slicing through image block mixing masks (mask) of these multiple images to obtain feature maps of the corresponding number of mixed images, so as to realize self-supervised learning of downstream tasks and improve the performance of the classification model.

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. For the convenience of description, only the parts related to the relevant inventions are shown in the drawings. In the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. That is to say, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Among them, the present invention uses a flow chart to illustrate the operations performed by the system according to an embodiment of the present invention. It should be understood that the previous or subsequent operations are not necessarily performed accurately in order. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0057] Reference Figure 1 , which is a flow chart of an optional embodiment 1 of the eye image classification method based on self-supervised learning proposed in this application. The method can be applicable to electronic devices, which can be servers such as independent physical servers, server clusters integrated with multiple physical servers, or cloud servers with cloud computing capabilities, and / or terminal devices with certain data processing capabilities, such as smart medical devices, robots, desktop computers, etc. Of course, in some scenarios, the terminal device can also interact with the server to implement the eye image classification method based on self-supervised learning proposed in this application. Based on this, Figure 1 As shown, the eye image classification method based on self-supervised learning proposed in this embodiment may include:

[0058] Step S11, obtaining an eye image to be identified;

[0059] In practical applications of the present application, a slit lamp or other medical equipment may be used to capture images of the patient's eyes to obtain an eye image to be identified that includes eye features. The present application does not impose any restrictions on the method for acquiring the eye image.

[0060] It should be noted that, if the above-mentioned medical equipment has data processing capabilities, the medical equipment can be used as an electronic device to execute the eye image classification method based on self-supervised learning proposed in this application; if the medical equipment does not have data processing capabilities, the eye image to be identified collected from a certain eye can be sent to an electronic device (such as a terminal or server, etc.) through wired communication or wireless communication, so that the electronic device executes the eye image classification method based on self-supervised learning proposed in this application to obtain the corresponding category results, such as determining whether the eye image to be identified belongs to the category of ocular surface diseases, and even determining which disease the image category corresponds to.

[0061] Step S12, processing the eye image to be identified by using the eye image classification model obtained based on self-supervised learning to obtain category prediction information; wherein the eye image classification model is obtained by transferring the pre-trained supervision task model to the eye image classification task model, and then optimizing the model parameters using the sample image, and the supervision task model is obtained by performing supervised learning on the predicted positions of the image blocks of a group of mixed images, and the group of mixed images is obtained by mixing a group of sample images in blocks;

[0062] Combined with the above description of the technical solution of the present application, the present application uses a large number of unlabeled sample images and adopts a self-supervised learning method to obtain a classification model for eye image classification tasks (i.e., downstream tasks), i.e., an eye image classification model. In the self-supervised learning process, a mixed image feature map obtained by mixing the sample images in blocks is used. In the network training process of the upstream task, i.e., the supervised task, the original position of the corresponding feature block is restored by predicting which sample image each image block in the mixed image belongs to, i.e., reconstructing the feature map for the original sample image. The implementation process is not described in detail in this embodiment.

[0063] For the supervised task model learned by the upstream task, the pre-trained model of the downstream task is used through transfer learning, and the classifier required by the downstream task is combined as the output layer. After fine-tuning the model parameters, the eye image classification model is obtained. This application does not elaborate on the implementation process of transfer learning and model parameter fine-tuning.

[0064] In this way, in the application scenario of executing downstream tasks, after obtaining the eye image to be identified, the trained eye image classification model can be directly input to obtain the corresponding category prediction information, such as the predicted probability or score of the eye image to be identified belonging to different categories, and even the pathological feature areas belonging to the corresponding category in the eye image to be identified can be identified. This application does not describe the classification prediction implementation process in detail.

[0065] Step S13: using the category prediction information, outputting the category result of the eye image to be identified.

[0066] Following the above analysis, if the category prediction information is a prediction value, the category corresponding to the maximum prediction value can be determined as the target category of the eye image to be identified. If needed, the lesion feature area of ​​the corresponding category can also be included, thereby constituting the category result of the eye image to be identified. The present application does not limit the content of the category result, and can be determined in combination with the model structure of the downstream task in the above self-supervised learning process, including but not limited to the implementation method described above.

[0067] Optionally, when the electronic device is a medical device, it can send the eye image to be identified and its category results to a preset terminal, or upload it to a server for storage for ophthalmologists to review, or send it to a printing device for printout, so as to assist the doctor to more accurately determine whether the eye to be identified has an ocular surface disease, or even what kind of ocular surface disease it has, i.e., the target category, etc., laying the foundation for providing patients with accurate treatment plans.

[0068] It can be seen that the present application trains the upstream task model, i.e., the supervision task model, by using the feature maps of a group of mixed images obtained after block mixing of a group of unlabeled sample images, and restores the feature maps of the original sample images, i.e., the unmixed feature maps, thereby improving the efficiency and accuracy of supervision information mining. Afterwards, the supervision task model is transfer-learned as a pre-trained model for the downstream task, i.e., the eye image classification task. After fine-tuning the model parameters, the corresponding eye image classification model is obtained. There is no need to label a large number of sample images, which greatly saves the cost of manual labeling and improves the model learning efficiency.

[0069] Reference Figure 2 , which is a flow chart of an optional embodiment 2 of the eye image classification method based on self-supervised learning proposed in this application. This embodiment can describe the process of obtaining the eye image classification model obtained based on self-supervised learning above, such as Figure 2 As shown, the acquisition process may include but is not limited to the following steps:

[0070] Step S21, obtaining an unlabeled training data set;

[0071] As analyzed above, the training data set can be clinical images directly collected by medical equipment. In order to improve the efficiency and reliability of subsequent model training, clinical images such as those taken by slit lamps can be first desensitized and cleaned to eliminate variables that are irrelevant to the eye image classification task and have excessive missing variables, as well as clinical images with obvious errors or data missing. This application does not limit the method for accurately implementing data, which can be determined according to the circumstances.

[0072] Step S22, randomly selecting a group of sample images from the training data set; the group of sample images includes multiple sample images;

[0073] Step S23, dividing each sample image in the group of sample images into the same number of image blocks;

[0074] In the preprocessing stage before performing self-supervised learning, refer to Figure 3The preprocessing flow diagram shown can select a group of sample images from the training data set, such as randomly selecting 4 clinical images as a group of sample images, etc. The present application does not limit the number of sample images contained in a group of sample images and the selection implementation method. The present application only takes the preprocessing process of 4 sample images as a group as an example to illustrate. The implementation methods of the preprocessing processes for other sample image groups are similar and will not be described in detail one by one in this embodiment.

[0075] For each sample image included in a set of sample images, it can be divided into multiple regular image blocks, etc. The present application does not limit the size of each image block, and can be determined in combination with the size of the sample image so that the image blocks divided from a sample image have the same size. Among them, if the sizes of different sample images are the same, they can be divided into blocks with the same specifications to obtain image blocks of the same size, so that the image blocks of different sample images can be mixed and processed later.

[0076] Step S24, based on the number of image blocks obtained by segmenting each sample image, obtaining an image block mixed mask for the group of sample images; the image block mixed mask is composed of image identifiers of the plurality of sample images;

[0077] Step S25, performing mixing processing on the image blocks in the group of sample images based on the image block mixing mask to obtain a corresponding group of mixed images;

[0078] In an embodiment of the present application, the image block mixing mask mask can represent the mixing rules for each image block contained in an original set of sample images. The present application needs to mix the image blocks belonging to different sample images in the set of sample images so that multiple image blocks in the obtained mixed image can come from different sample images. The present application does not limit the method for generating the image block mixing mask.

[0079] As analyzed above, in order to conveniently describe which sample image the segmented image block belongs to, a one-to-one corresponding image identifier can be configured for multiple sample images in a set of sample images, such as Figure 3 The 0, 1, 2 and 3 shown in the figure represent each sample image in a set of sample images, and then, the number of image blocks segmented from each sample image or their layout is combined, such as Figure 3 As shown in the figure, a 4*4 image block, that is, a sample image contains 4 rows and 4 columns of image blocks, randomly generates image block mixed mask, such as Figure 3 Bottom left image, etc.

[0080] In the image block mixing mask, the image identifier at each position contained therein can indicate that the image block at that position of the corresponding mixed image can come from the image block at the corresponding position of the image identifier, such as Figure 3In , the element 3 in the first row and first column of the image block mixing mask mask represents a mixing rule that the image blocks in the first row and first column of the sample image with image identification 3 are determined as the image blocks in the first row and first column of the mixed image, that is, the image blocks in the first row and first column of the mixed image come from the sample image with image identification 3; the element 2 in the first row and second column of the image block mixing mask mask represents a mixing rule that the image blocks in the first row and second column of the sample image with image identification 2 are determined as the image blocks in the first row and second column of the mixed image.

[0081] Based on this, the present application can directly perform mixing processing on the image blocks of each sample image in the same group of sample images according to the image block mixing mask mask to obtain a corresponding group of mixed images. It should be noted that the mixing processing method for each image block contained in a group of sample images includes but is not limited to the implementation method described above. Optionally, each image block contained in each mixed image in a group of mixed images can come from at least one image library of each sample image in the corresponding group of sample images, that is, the sub-mixing mask for each mixed image in the above-generated image block mixing mask mask can include all image identifiers corresponding to a group of sample images, but is not limited to this mixing method.

[0082] Step S26, inputting the group of mixed images into the encoding network for feature extraction to obtain a corresponding feature map;

[0083] In an embodiment of the present application, the encoding network, that is, the feature extraction network, can adopt three convolutional layers of a convolutional neural network CNN, that is, three layers of CNN convolutions of different sizes, and input the above-mentioned set of mixed images into the encoding network for feature extraction, and output the corresponding feature map to facilitate subsequent self-supervised learning.

[0084] Optional, such as Figure 3 As shown, the size of each convolution layer included in the coding network can be: 48×11×11, 128×5×5, 192×3×3, respectively. Through convolution layers of different sizes, different dimensional features of a set of mixed images can be extracted to enrich the feature information of the obtained feature map. Of course, the size of each convolution layer included in the coding network can also be flexibly adjusted according to actual needs. This application does not limit the size of the convolution kernel of the convolution layer, which can be determined according to the situation.

[0085] Step S27, obtaining an image block restoration mask for a group of mixed images based on the image block mixing mask;

[0086] Step S28, inputting the feature map and the image block restoration mask into the decoding network, and obtaining the auxiliary task model by performing supervised learning on the image identifier of the sample image to which the predicted corresponding image block belongs;

[0087] Combination Figure 4 The self-supervised learning process diagram shown in the figure has a self-supervised learning module divided into upstream tasks (i.e., supervision tasks) and downstream tasks (such as the eye image classification task of the present application). The mixed unlabeled image obtained above, i.e., a feature map of a group of mixed images, can be input into the upstream task, and the sample image to which the image block corresponding to each feature block in the feature map belongs is predicted, that is, the feature blocks of the feature map are unmixed, and the original positions of the feature blocks are restored. In contrast to predicting the feature map corresponding to a group of sample images, the training and learning of the decoding network used for unmixing the feature maps is realized, thereby obtaining a representation valuable for the downstream tasks.

[0088] Among them, in the above-mentioned training and learning process, the image block restoration mask mask obtained based on the image block mixing mask mask can be determined as a pseudo-label to realize the loss calculation with the prediction result, and obtain the corresponding position prediction loss. By minimizing the position prediction loss, the decoding network is trained to obtain an auxiliary task model for downstream tasks. The implementation process is not described in detail in this embodiment.

[0089] Step S29, transferring and learning the auxiliary task model to be a pre-trained model of a downstream eye image classification task model; the output layer of the downstream eye image classification task model is a classifier;

[0090] Step S210, fine-tuning the model parameters of the pre-trained model using the sample image to obtain an eye image classification model.

[0091] For the auxiliary task model obtained from the upstream task, it can be transfer learned as a pre-trained model for the downstream task, combined with the classifier of the downstream task output layer to form a downstream task model. After that, the model parameters can be fine-tuned using unlabeled sample images to quickly obtain the required eye image classification model. This application does not impose any restrictions on the implementation method of transfer learning and model parameter fine-tuning, which can be determined according to the situation.

[0092] Reference Figure 5 , is a flow chart of an optional embodiment 3 of the eye image classification method based on self-supervised learning proposed in this application. This embodiment can describe an optional refinement of the self-supervised learning of the eye image classification model described above, such as Figure 5 As shown, the method may include:

[0093] Step S51, obtaining a group of sample images; the group of sample images includes a plurality of sample images;

[0094] Combined with the above method of obtaining the training data set, assuming that the training data set containing N sample images can be expressed as That is, {x 1 、x2 ,……,x N}, x u It can represent the uth sample image in the training data set, u∈(1, 2, ..., N), and this application does not limit the value of N. A group of sample images randomly selected therefrom can also be represented in this way, and N can represent the number of sample images contained in the group of sample images, such as N=4 in the above example.

[0095] Step S52, dividing each sample image in the group of sample images into the same number of image blocks;

[0096] Step S53, determining the image identifier corresponding to each sample image in the group of sample images;

[0097] Step S54, randomly generating a sub-mixing mask for each sample image based on the number of image blocks obtained by segmenting the sample image; the number of image identifiers contained in the sub-mixing mask is the same as the number of image blocks into which a sample image is segmented;

[0098] Step S55, forming an image block mixing mask for the group of sample images from the sub-mixing masks;

[0099] Step S56, according to the image identifiers at different positions in each sub-mixed mask, a mixed image is formed by segmenting the image blocks at corresponding positions of the sample image with the corresponding image identifier;

[0100] Combined with the description of the image block mixed mask mask related content in the above embodiment, and Figure 3 As shown in the preprocessing flow diagram, if four randomly selected sample images are taken as a group, four sub-mixed masks can be obtained according to the method described above, and their position layout in the image block mixed mask is consistent with the position layout relationship of multiple sample images in a group of sample images, such as Figure 3 shown.

[0101] In this way, in the mixing mask of the group of sample images and the corresponding image blocks, the image blocks contained in the group of sample images can be mixed according to the sub-mixing mask to obtain a mixed image with the same position layout, and the corresponding number of mixed images obtained constitute a group of mixed images with the same position layout relationship.

[0102] Step S57, inputting the group of mixed images into the encoding network for feature extraction to obtain a corresponding feature map;

[0103] Step S58, obtaining an image block restoration mask for a group of mixed images based on the image block mixing mask;

[0104] As analyzed above, in the embodiment of the present application, the image block mixing mask and the image block restoration mask correspond to each other. The former is used to perform image block mixing on an original set of sample images to obtain a corresponding set of mixed images, and the latter is used to restore the positions of the image blocks in the set of mixed images to obtain the original sample images. In the embodiment of the present application, the latter demixing process is performed on the feature map of the set of mixed images, and the feature map is composed of feature blocks corresponding to each image block of the set of mixed images. The image block restoration mask can be used to restore the original position of the feature blocks to obtain the feature map of the original set of sample images, such as Figure 4 The upper right feature map shown.

[0105] Among them, since the image block restoration mask is directly generated based on the image block mixing mask, if the mixing is directly performed based on it, the feature map of the original sample image can be accurately obtained, that is, the original positions of all feature blocks are restored. Therefore, the present application can use the image block restoration mask as a pseudo-label to enable the subsequent network to predict which sample image each feature block in the feature map of the mixed image belongs to, and perform loss calculation with the corresponding predicted restoration mask to achieve continuous self-supervised training and learning of the network.

[0106] It can be seen that for different image block mixing masks, the generated image block restoration masks can be different, including but not limited to the following Figure 3 and Figure 4 As analyzed above, if m(·) represents the image block (i.e., image slice) mixing function, then by mixing the image blocks of each sample image, a set of mixed images can be obtained, i.e., m(x u ) → C u , after the feature extraction of the convolutional neural network f(·) composed of three layers of CNN convolution, the corresponding feature map is obtained, that is,

[0107] Step S59, performing semantic analysis on each feature block included in the feature map based on the decoding network, and identifying the predicted image identifier of the sample image to which the corresponding image block belongs;

[0108] After the above-mentioned image block mixing processing, although the edges or parts of objects in the mixed image and its feature map are difficult to identify by mixing image blocks of different sample images, this mixing method does not lose object information. The decoding network can more accurately and efficiently remove the mixing and restore the original position of the feature.

[0109] Among them, after the above-mentioned image block mixing, since the mixed image blocks will produce severe occlusion, the corresponding feature blocks in the feature map can only contain their local information. In order to improve the accuracy of demixing and identify which sample image the image blocks in the mixed image come from, the decoding network of the present application can focus on high-level semantic information in the feature map of the mixed image during the input analysis process, such as texture features or edge information, that is, to better learn the semantic information of the image and improve the accuracy of demixing prediction.

[0110] In the embodiment of the present application, the above decoding network can be a convolutional neural network, i.e., a ConvNet network, which can use a deep residual network ResNet-50 as a backbone network to achieve the above position prediction. The specific network structure of the decoding network is not described in detail in the present application. If u(·) represents the unmixing function, the mixing can be removed by step S59, i.e.

[0111] Step S510, based on the cross entropy loss function, the predicted image identifier and the image identifier of the image block corresponding to the image block restoration mask are calculated to obtain the position prediction loss of the corresponding image block;

[0112] Step S511, training the decoding network by minimizing the position prediction loss to obtain an auxiliary task model for the eye image classification task;

[0113] The feature map of the mixed image and the image block restoration mask are input into the decoding network. After continuous training and learning, it is predicted which sample image the corresponding image block belongs to. In the process of restoring the original position of the feature, the cross entropy loss function can be used for each training to calculate the corresponding position prediction loss L u , and accordingly adjust the network parameters to reduce the position prediction loss until the position prediction loss converges or reaches the preset training times and other training termination conditions, and obtain the upstream task model, that is, the auxiliary task model.

[0114] Among them, based on the cross entropy loss function CrossEntropy(), the position prediction loss L is calculated u The formula can be:

[0115]

[0116] In formula (1), if f t (·) represents the upstream task model, P u It can represent the predicted value that the image block belongs to the sample image (which can be represented by the predicted image identifier), that is, It can represent the image block recovery mask, i.e., the pseudo label. The cross entropy loss calculation process of the upstream task model in this application is No further details.

[0117] Step S512, transferring and learning the auxiliary task model to be a pre-trained model of a downstream eye image classification task model; the output layer of the downstream eye image classification task model is a classifier;

[0118] Step S513, fine-tuning the model parameters of the pre-trained model using the sample image to obtain an eye image classification model.

[0119] As analyzed above, this application can transfer the upstream task model f t (·) (i.e., supervised task model) transfer learning to the pre-trained model of the downstream task model, combined with the classifier of the output layer, to obtain the downstream task model f s (·), that is, f s (·)←f t (·), that is, after fine-tuning the model parameters, the eye image classification model is obtained. The implementation process is not described in detail in this application.

[0120] Reference Figure 6 , is a schematic diagram of the structure of an optional embodiment of the eye image classification device based on self-supervised learning proposed in this application, such as Figure 6 As shown, the device may include:

[0121] The eye image to be identified obtaining module 61 is used to obtain the eye image to be identified;

[0122] The category prediction information obtaining module 62 is used to process the eye image to be identified by using the eye image classification model obtained based on self-supervised learning to obtain category prediction information; wherein the eye image classification model is obtained by transferring the pre-trained supervision task model to the eye image classification task model and fine-tuning the model parameters using the sample image, and the supervision task model is obtained by performing supervised learning on the predicted positions of the image blocks of a group of mixed images, and the group of mixed images is obtained by mixing a group of sample images in blocks;

[0123] The category result output module 63 is used to output the category result of the eye image to be identified by using the category prediction information.

[0124] Optional, such as Figure 7 As shown, a mixing module for mixing a group of sample images into blocks to obtain a group of mixed images includes:

[0125] The sample image group obtaining unit 64 is used to obtain a group of sample images; the group of sample images includes a plurality of sample images;

[0126] A segmentation unit 65, configured to segment each of the sample images in the set of sample images into image blocks of the same number;

[0127] An image block mixed mask obtaining unit 66 is used to obtain an image block mixed mask for the group of sample images based on the number of image blocks obtained by segmenting each of the sample images; the image block mixed mask is composed of image identifiers of the respective sample images;

[0128] The mixing unit 67 is used to perform mixing processing on the image blocks in the set of sample images based on the image block mixing mask to obtain a corresponding set of mixed images.

[0129] Optionally, the image block mixed mask obtaining unit 66 may include:

[0130] An image identification determination unit, used to determine an image identification corresponding to each sample image in the group of sample images;

[0131] A sub-mixing mask generating unit, configured to randomly generate a sub-mixing mask for each sample image based on the number of image blocks obtained by segmenting the sample image; the number of image identifiers included in the sub-mixing mask is the same as the number of image blocks into which one sample image is segmented;

[0132] The image block mixing mask forming unit is used to form an image block mixing mask for the group of sample images by using the sub-mixing masks.

[0133] Optionally, the mixing unit 67 may include:

[0134] The first processing unit is used to form a mixed image from the image blocks at corresponding positions after the sample image with the corresponding image identifier is segmented according to the image identifier at different positions in each of the sub-mixed masks.

[0135] In some other embodiments, Figure 7 As shown, the supervised task model obtaining module for obtaining the supervised task model by performing supervised learning on the predicted positions of the image blocks of a group of mixed images includes:

[0136] A feature map obtaining unit 68 is used to input the set of mixed images into a coding network for feature extraction to obtain a corresponding feature map;

[0137] An image block restoration mask obtaining unit 69, configured to obtain an image block restoration mask for the group of mixed images based on the image block mixing mask;

[0138] The auxiliary task model obtaining unit 610 is used to input the feature map and the image block restoration mask into a decoding network, and obtain an auxiliary task model by performing supervised learning on the image identifier of the sample image to which the corresponding image block is predicted.

[0139] Optionally, the auxiliary task model obtaining unit 610 may include:

[0140] A predicted image identifier recognition unit, configured to perform semantic analysis on each feature block included in the feature map based on a decoding network, and identify a predicted image identifier of the sample image to which the corresponding image block belongs;

[0141] A position prediction loss obtaining unit, configured to calculate the predicted image identifier and the image identifier of the image block corresponding to the image block restoration mask based on a cross entropy loss function to obtain a position prediction loss corresponding to the image block;

[0142] The network training unit is used to train the decoding network by minimizing the position prediction loss to obtain an auxiliary task model for the eye image classification task.

[0143] Optionally, the eye image classification model obtaining module may include:

[0144] A transfer learning unit, used for transfer learning of the auxiliary task model to a pre-trained model of a downstream eye image classification task model; the output layer of the downstream eye image classification task model is a classifier;

[0145] A fine-tuning unit is used to use the sample image to fine-tune the model parameters of the pre-trained model to obtain an eye image classification model.

[0146] It should be noted that the various modules, units, etc. in the above-mentioned device embodiments can be stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to implement the corresponding functions. For the functions implemented by each program module and its combination, as well as the technical effects achieved, please refer to the description of the corresponding parts of the above-mentioned method embodiments, which will not be repeated in this embodiment.

[0147] The present invention also provides a computer-readable storage medium on which a computer program can be stored. The computer program can be called and loaded by a processor to implement the various steps of the eye image classification method based on self-supervised learning described in the above embodiment. The specific implementation process can refer to the description of the corresponding part of the above embodiment, which will not be repeated in this embodiment.

[0148] Reference Figure 8 , is a schematic diagram of the hardware structure of an optional example of an electronic device applicable to the eye image classification method based on self-supervised learning proposed in this application, such as Figure 8 As shown, the electronic device may include: a communication connection port 81 and a processor 82, wherein:

[0149] The number of the communication connection port 81 and the processor 82 can be at least one, and the communication connection port 81 and the processor 82 can be connected to a communication bus to achieve data interaction between each other through the communication bus. The specific implementation process can be determined according to the requirements of the specific application scenario, and the present invention will not elaborate on it.

[0150] The communication connection port 81 may include a communication interface that can realize data interaction by using a wireless communication network, such as a WIFI module, a 5G / 6G (fifth generation mobile communication network / sixth generation mobile communication network) module, a GPRS module, etc., so as to realize wireless communication between the electronic device and other devices, such as obtaining an eye image to be identified, outputting category results, etc.; of course, the communication connection port 81 can also be a data interface for realizing wired communication, so that the electronic device can be connected to other devices through a corresponding type of data cable to realize data transmission between different devices.

[0151] It should be understood that the above-mentioned communication connection port 81 includes a communication interface for realizing data interaction between internal components of the electronic device, such as a USB interface, a serial / parallel port, a multimedia transmission interface, a universal asynchronous receiver / transmitter (UART) interface, etc. The present application does not limit the type and number of the communication connection port 81.

[0152] In an embodiment of the present application, the processor 82 can load and execute stored computer instructions to implement the various steps of the eye image classification method based on self-supervised learning proposed in any of the above method embodiments of the present application. The specific implementation process can refer to the description of the corresponding part of the corresponding embodiment above and will not be repeated here.

[0153] In practical applications, the processor 82 may include at least one storage unit and a processing unit, wherein the storage unit may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device. The processing unit may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, etc.

[0154] Optionally, when the processor 82 has storage space inside, such as the processors listed above, it is not necessary to integrate or connect the memories listed above, and the eye image classification method based on self-supervised learning proposed in the present application can be directly implemented.

[0155] It should be understood that Figure 8 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiment of the present invention. In practical applications, the electronic device may include Figure 8 More components shown, or a combination of certain components, can be determined based on the product type of the electronic device. When the electronic device is a terminal device, the electronic device may also include: an image acquisition component, a display, a speaker and at least one output component. It may also include a pickup, a function button and at least one input component as needed to meet different input operations on the electronic device, as well as a sensor module containing multiple sensors, a power supply component, etc., which can be determined based on the application scenario and are not listed one by one in this application.

[0156] Finally, it should be noted that the "system", "device", "unit" and / or "module" used in the present invention is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0157] As shown in the present invention and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0158] Among them, in the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" refers to two or more than two. The following terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0159] In addition, the various embodiments in this specification are described in a progressive or parallel manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the devices and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0160] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for eye image classification based on self-supervised learning, It is characterized in that The method comprises: Obtaining an eye image to be identified; The eye image to be identified is processed by an eye image classification model obtained based on self-supervised learning to obtain category prediction information; wherein the eye image classification model is obtained by transferring the pre-trained supervised task model to the eye image classification task model and fine-tuning the model parameters using sample images, and the supervised task model is obtained by supervised learning of the predicted positions of the image blocks of a group of mixed images, and the group of mixed images is obtained by mixing a group of sample images in blocks; The category prediction information is used to output the category result of the eye image to be identified.

2. The method according to claim 1, It is characterized in that The step of mixing a group of sample images into blocks to obtain a group of mixed images comprises: Obtaining a set of sample images; the set of sample images includes a plurality of sample images; Dividing each of the sample images in the set of sample images into the same number of image blocks; Based on the number of image blocks obtained by segmenting each of the sample images, an image block mixing mask for the group of sample images is obtained; the image block mixing mask is composed of image identifiers of the plurality of sample images; Based on the image block mixing mask, the image blocks in the set of sample images are mixed to obtain a corresponding set of mixed images.

3. The method according to claim 2, It is characterized in that The step of obtaining an image block mixed mask for the group of sample images based on the number of image blocks obtained by segmenting each sample image comprises: Determine an image identifier corresponding to each sample image in the set of sample images; Based on the number of image blocks obtained by segmenting the sample image, a sub-mixing mask for each sample image is randomly generated; the number of image identifiers included in the sub-mixing mask is the same as the number of image blocks into which one sample image is segmented; An image block blending mask for the set of sample images is constructed by the sub-blending masks.

4. The method according to claim 3, It is characterized in that The step of performing mixing processing on the image blocks in the set of sample images based on the image block mixing mask to obtain a corresponding set of mixed images includes: According to the image identifiers at different positions in each of the sub-mixed masks, a mixed image is formed by the image blocks at corresponding positions after the sample image with the corresponding image identifier is segmented.

5. The method according to any one of claims 2 to 4, It is characterized in that The supervised learning is performed on the predicted positions of the image blocks of a group of mixed images to obtain a supervised task model, including: Inputting the set of mixed images into a coding network for feature extraction to obtain a corresponding feature map; Based on the image block mixing mask, obtaining an image block restoration mask for the set of mixed images; The feature map and the image block restoration mask are input into a decoding network, and supervised learning is performed on the image identifier of the sample image to which the predicted corresponding image block belongs, so as to obtain an auxiliary task model.

6. The method according to claim 5, It is characterized in that The step of inputting the feature map and the image block restoration mask into a decoding network and performing supervised learning on the image identifier of the sample image to which the predicted corresponding image block belongs to obtain an auxiliary task model comprises: Performing semantic analysis on each feature block included in the feature map based on a decoding network to identify a predicted image identifier of the sample image to which the corresponding image block belongs; Based on a cross entropy loss function, the predicted image identifier and the image identifier of the image block corresponding to the image block restoration mask are calculated to obtain a position prediction loss corresponding to the image block; By minimizing the position prediction loss, the decoding network is trained to obtain an auxiliary task model for the eye image classification task.

7. The method according to claim 5, It is characterized in that After the pre-trained supervised task model is transferred to the eye image classification task model, the model parameters are fine-tuned using sample images to obtain the eye image classification model, including: Transfer learning the auxiliary task model to a pre-trained model of a downstream eye image classification task model; the output layer of the downstream eye image classification task model is a classifier; The sample images are used to fine-tune the model parameters of the pre-trained model to obtain an eye image classification model.

8. The method according to claim 7, It is characterized in that The classifier is a nonlinear classifier consisting of three fully connected layers; The encoding network consists of three convolutional neural network (CNN) layers; The decoding network is a convolutional neural network ConvNet.

9. An eye image classification device based on self-supervised learning, It is characterized in that The device comprises: The module for obtaining the eye image to be identified is used to obtain the eye image to be identified; A category prediction information obtaining module, used to process the eye image to be identified through an eye image classification model obtained based on self-supervised learning to obtain category prediction information; wherein the eye image classification model is obtained by transferring the pre-trained supervision task model to the eye image classification task model and fine-tuning the model parameters using sample images, and the supervision task model is obtained by supervised learning of the predicted positions of the image blocks of a group of mixed images, and the group of mixed images is obtained by mixing a group of sample images in blocks; The category result output module is used to output the category result of the eye image to be identified by using the category prediction information.

10. An electronic device, It is characterized in that The electronic device comprises: A communication connection port, used for receiving an eye image to be identified; The processor is used to load and execute the computer to implement the following steps: Obtaining an eye image to be identified; The eye image to be identified is processed by an eye image classification model obtained based on self-supervised learning to obtain category prediction information; wherein the eye image classification model is obtained by transferring the pre-trained supervised task model to the eye image classification task model and fine-tuning the model parameters using sample images, and the supervised task model is obtained by supervised learning of the predicted positions of the image blocks of a group of mixed images, and the group of mixed images is obtained by mixing a group of sample images in blocks; The category prediction information is used to output the category result of the eye image to be identified.

Images